sist2
sist2 (Simple incremental search tool)
Warning: sist2 is in early development
Features
- Fast, low memory usage, multi-threaded
- Mobile-friendly Web interface
- Portable (all its features are packaged in a single executable)
- Extracts text from common file types *
- Generates thumbnails *
- Incremental scanning
- Automatic tagging from file attributes via user scripts
- Recursive scan inside archive files **
- OCR support with tesseract ***
- Stats page & disk utilisation visualization
* See format support
** See Archive files
*** See OCR
Getting Started
-
Have an Elasticsearch (>= 6.X.X) instance running
- Download from official website
- (or) Run using docker:
docker run -d --name es1 --net sist2_net -p 9200:9200 \ -e "discovery.type=single-node" elasticsearch:7.5.2
- (or) Run using docker-compose:
elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2 environment: - discovery.type=single-node - "ES_JAVA_OPTS=-Xms1G -Xmx2G"
-
Download sist2 executable
- Download the latest sist2 release *
- (or) Download a development snapshot (Not recommended!)
- (or)
docker pull simon987/sist2:latest
-
See Usage guide
* Windows users: sist2 runs under WSL
Example usage
See Usage guide for more details
- Scan a directory:
sist2 scan ~/Documents -o ./docs_idx
- Push index to Elasticsearch:
sist2 index ./docs_idx
- Start web interface:
sist2 web ./docs_idx
Format support
File type | Library | Content | Thumbnail | Metadata |
---|---|---|---|---|
pdf,xps,cbz,cbr,fb2,epub | MuPDF | text+ocr | yes, png |
title |
audio/* |
ffmpeg | - | yes, jpeg |
ID3 tags |
video/* |
ffmpeg | - | yes, jpeg |
title, comment, artist |
image/* |
ffmpeg | - | yes, jpeg |
Common EXIF tags |
ttf,ttc,cff,woff,fnt,otf | Freetype2 | - | yes, bmp |
Name & style |
text/plain |
(none) | yes | no | - |
html, xml | (none) | yes | no | - |
tar, zip, rar, 7z, ar ... | Libarchive | yes* | - | no |
docx, xlsx, pptx | (none) | yes | no | creator, modified_by, title |
mobi, azw, azw3 | libmobi | yes | no | author, title |
* See Archive files
Archive files
sist2 will scan files stored into archive files (zip, tar, 7z...) as if they were directly in the file system. Recursive (archives inside archives) scan is also supported.
Limitations:
- Parsing media files with formats that require
seek (e.g.
.gif
,.mp4
w/ fragmented metadata etc.) is not supported. - Archive files are scanned sequentially, by a single thread. On systems where sist2 is not I/O bound, scans might be faster when larger archives are split into smaller parts.
To check if a media file can be parsed without seek, execute cat file.mp4 | ffprobe -
OCR
You can enable OCR support for pdf,xps,cbz,cbr,fb2,epub file types with the
--ocr <lang>
option. Download the language data files with your
package manager (apt install tesseract-ocr-eng
) or directly from Github.
The simon987/sist2
image comes with common languages
(hin, jpn, eng, fra, rus, spa) pre-installed.
Examples
sist2 scan --ocr jpn ~/Books/Manga/
sist2 scan --ocr eng ~/Books/Textbooks/
Build from source
You can compile sist2 by yourself if you don't want to use the pre-compiled binaries (GCC 7+ required).
-
Install compile-time dependencies
vcpkg install lmdb cjson glib libarchive[core,bzip2,libxml2,lz4,lzma,lzo] pthread tesseract libxml2 ffmpeg zstd gtest mongoose libuuid libmagic
-
Build
git clone --recursive https://github.com/simon987/sist2/ cmake -D <VCPKG_ROOT>/scripts/buildsystems/vcpkg.cmake . make