608B71FC-006A-4934-A643-7D9BA9340450Blog

Modules

blog__author-img
Michał Czapracki
CEO at Scramjet, Data Streaming Expert.
23F2E8CD-3026-46A5-86CC-D13114F7176E425AE875-B1A1-4EA1-8529-075D08DA0BB1

16 January 2019

Scramjet modules are node modules that allow you to easily reuse a chain of transforms and even publish it for others to use. In this article we'll drill down on ways to write and use modules.

Module usage

Let's start with absolute basics, how modules are used based on scramjet-hello-module:

A module needs to be installed with this simple shell command:


_10
npm i scramjet-hello-module

Then we use it by passing it to the use method of any Scramjet stream.


_10
DataStream.from(someSource).use("scramjet-hello-module", argument1);

Modules will work also on any standard node.js stream:


_10
import hello from "scramjet-hello-module";
_10
_10
hello(myStream, argument1); // returns a new stream

You can also use modules that are not published in NPM or in node_modules:


_10
stream.use("./my-parse-module"); // will look for a file in the same directory as the file where it's used.
_10
stream.use("/path/to/some/module"); // will look for a certain path
_10
stream.use("my-module/submodule"); // will in node_modules/my-module/submodule.js

Finally sometimes you may want to use multiple modules:


_10
DataStream.from(someSource).use("./find-names").use("scramjet-hello-module", argument1);
_10
// ... and so on.

Developing a module

As an example we'll use a module that converts the stream entries to numbers and filters out non-numbers.

Basics first - it's all in a simple file:


_10
module.exports = require("scramjet").createTransformModule(stream =>
_10
stream.map(x => parseFloat(x)).filter(x => isNaN(x))
_10
);

If you save it as ./only-numbers you can use it in your code to filter out non-numbers from any of your streams in the specific project you put your file in. That's it, you're done, use it and have fun...

But what if you need to use this module in another project?

Fair question. Why not make an actual node module out of it - what you need is a package.json and somewhere to store it - a git repo works well.

Create a directory and put your only-numbers.js file.


_12
# create your local git repo
_12
$ git init .
_12
_12
# add your remote (create it on github or wherever you like)
_12
$ git remote add origin [email protected]:yourname/yourmodule.git
_12
_12
# Answer some questions here and you'll get a nice package json.
_12
# make sure that the answer to "entry point: (index.js)" is "only-numbers.js"
_12
$ npm init
_12
_12
# Add commit and push
_12
$ git add . && git commit -m "Initial commit" && git push

You'll end up with a module in git that should have the following files:


_10
\
_10
|- .git/ # this is where your git files sit
_10
|- only-numbers.js # this is your actual scramjet module
_10
|- package.json # this is where your git files sit

Can I fork something to develop my module quicker?

Sure! Here's a repo made especially for you: signicode/scramjet-module-hello.

What if my module is good for public?

I congratulate you sir! Now this is fairly simple:


_10
npm publish

But please consider doing a couple things first:

  • Read the NPM guide on publishing packages - this will make you package stand out.
  • Consider adding some checks like Snyk so that you keep your package up to date.
Project co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.