POSTED 24 days ago

Getting started with IPFS-Store

Getting started with IPFS-Store "A Search engine for the distributed web"

IPFS-Store is a search engine opensource tool aiming to collect, store and index data on the IPFS network. This is a convenient solution for any applications requiring content discovery (conditional queries or full text search with fuzziness) in a set of IPFS files. This service also offers extra-features for you IPFS environment such as multi-pinning (replication), smart contract event listening (wip).

IPFS-Store can be deployed as a simple, scalable and configurable API and comes with client libraries (Java, JavaScript) to easily integrate it in an application.

A request requires the following the information:

  • a payload (JSON or multipart file) - stored on IPFS
  • some metadata (index fields) - indexed on a search engine alongside the payload IPFS hash


Start up IPFS-Store

We're gonna use Docker to run IPFS-Store and its dependencies because nowadays this is a very accessible and agnostic method for developers and devops to configure and run services.

Prerequisites

IPFS-Store depends of two components:

  • an IPFS node (go or js implementation)
  • a search engine (currently only ElasticSearch is supported)

Create a virtual network

Let's create a private Docker network to make our containers able to communicate together.

$ docker network create ipfs-store-network

Start IPFS daemon

Start an IPFS daemon to join the IPFS network and expore the port 4001 (peer2p networking) and 5001(API)

$ docker run -d --name ipfs -v /path/to/ipfs/export:/export -v /path/to/ipfs/data:/data/ipfs -p :4001:4001 -p :5001:5001 ipfs/go-ipfs:latest

Start ElasticSearch

Start ElasticSearch database used as a content indexer.

$ docker run -d --name elasticsearch -v /path/to/elasticsearch:/data/elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --net ipfs-store-network docker.elastic.co/elasticsearch/elasticsearch:6.5.0

Start IPFS-Store

Finally we can run IPFS-Store using the port 8040.

$ docker run --name ipfs_store -p 8040:8040 -e IPFS_HOST=ipfs -e ELASTIC_HOST=elasticsearch --net ipfs-store-network  gjeanmart/ipfs-store

A docker-compose file can also be used to start IPFS-Store and its dependencies.

To stop the containers, run $ docker stop ipfs elasticsearch ipfs_store.


Using IPFS-Store

Now we have IPFS-Store up and running, we can start publishing some content.

Publish a file and index metadata (title, date, author)

Publishing requires a content (file or JSON payload) and a set of metadata to enable content discovery. The function notably returns the index where this content is indexed, a unique identifier within the index and the content addressable IPFS hash.

$ curl -X POST ' http://localhost:8040/ipfs-store/raw/store_index' \
  -H 'content-type: multipart/form-data' \
  -F file=@/home/gjeanmart/book.pdf \
  -F 'request={"index":"books","index_fields":[{"name":"title","value":"book.pdf"},{"name":"author","value":"Gregoire Jeanmart"},{"name":"date_created","value":1518700549}]}'

{"index":"mydocs","id":"iPRcHGcBWSc03x36PSQw","hash":"QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u"}

Publish a JSON content and index metadata

$ curl -X POST 'http://localhost:8040/ipfs-store/json/store_index' \
  -H 'content-type: application/json' \
  -d '{
      "payload":  {
          "isbn": "9781593275846",
          "title": "Eloquent JavaScript, Second Edition",
          "subtitle": "A Modern Introduction to Programming",
          "author": "Marijn Haverbeke",
          "published": "2014-12-14T00:00:00.000Z",
          "publisher": "No Starch Press",
          "pages": 472
      },
      "index": "books",
      "content_type": "application/json",
      "index_fields": [
          {
            "name": "title",
            "value": "Eloquent JavaScript, Second Edition"
          },
          {
            "name": "subtitle",
            "value": "A Modern Introduction to Programming"
          },
          {
            "name": "author",
            "value": "Marijn Haverbeke"
          },
          {
            "name": "published",
            "value": "2014-12-14T00:00:00.000Z"
          }
      ]
  }'  

{"index":"books","id":"ifRtHGcBWSc03x36EiSz","hash":"QmY8166NpeN9VQRmkSKzzbBj3NN3vxcchSCxh1tY953WJ5"}

Fetch content

This function retrieves a document hosted on the IPFS network by its unique hash.

$  curl 'http://localhost:8040/ipfs-store/query/fetch/QmY8166NpeN9VQRmkSKzzbBj3NN3vxcchSCxh1tY953WJ5' 

{"isbn":"9781593275846","title":"Eloquent JavaScript, Second Edition","subtitle":"A Modern Introduction to Programming","author":"Marijn Haverbeke","published":"2014-12-14T00:00:00.000Z","publisher":"No Starch Press","pages":472}

Search content

A search returns a paginated list of metadata result including the IPFS hash of the document which can be loaded using the function above.

$ curl -X POST 'http://localhost:8040/ipfs-store/query/search?index=books' \
    -H 'content-type: application/json' \
    -d '{"query":[{"name":"title","operation":"full_text","value":"JavaSc"}]}'

{"content":[{"index":"books","id":"ivR0HGcBWSc03x36ZyTU","hash":"QmY8166NpeN9VQRmkSKzzbBj3NN3vxcchSCxh1tY953WJ5","content_type":"application/json","index_fields":[{"name":"__content_type","value":"application/json"},{"name":"author","value":"Marijn Haverbeke"},{"name":"subtitle","value":"A Modern Introduction to Programming"},{"name":"__hash","value":"QmY8166NpeN9VQRmkSKzzbBj3NN3vxcchSCxh1tY953WJ5"},{"name":"published","value":"2014-12-14T00:00:00.000Z"},{"name":"title","value":"Eloquent JavaScript, Second Edition"}]}],"last":true,"totalElements":2,"totalPages":1,"first":true,"numberOfElements":2,"sort":null,"size":20,"number":0} 

Learn more

IPFS-Store is still at a very early stage but can already be used by your Dapp to publish data on the IPFS network and maintain a search functionality.

If you want to learn more, you can explore the different configuration tweaks ready to use and the full documentation of the API.

If you want to be involve, we are currently working on the following tasks:

  • Implementation of a JavaScript client (wip)
  • Smart Contract Event listener (wip)
  • PGP Encryption
  • Access control
  • Other search engine implementation
  • Improving documentation and content in general (tutorials, clients documentation, etc…)
Outline
  • Getting started with IPFS-Store "A Search engine for the distributed web"

  • Start up IPFS-Store

  • Using IPFS-Store

  • Learn more