Introduction to Sciline

Introduction to Sciline#

Data processing workflow management tool.

scipp.github.io/sciline/

It is worth taking your time going through this notebook, do not try to rush it.

Sciline is an open-source library developed by ESS for managing and visualizing data processing workflows (sometimes called “pipelines”).

It defines workflows as directed acyclic graphs (DAGs) where the nodes are inputs, intermediate results, or final results, and edges are dependencies between the nodes.

This has some benefits:

Any (named) intermediate result in the pipeline can be computed.
The dependencies between intermediate results can be visualized.
Implementations of intermediate result can be replaced.
Results that are expensive to compute can be cached.
Certainty that the computed result has not been corrupted by running jupyter cells out of order.

Terminology#

A Sciline workflow (or “pipeline”) is defined by a set of transformations or providers that specify what inputs are needed to compute one specific output quantity.

The input and output quantities of the providers are called domain types.

A provider is a python function with type annotations:

# Example provider
def load_run(
    run_number: RunNumber,
    proposal_number: ProposalNumber,
    data_dir_path: DataPath,
) -> LoadedNexusData:

    filename = f'{proposal_number}_{run_number:06d}.hdf'
    path = os.path.join(data_dir_path, filename)

    data = scippnexus.load(path)
        
    return data

In the above example, to compute the LoadedNexusData quantity the RunNumber, ProposalNumber and the DataPath quantities are needed.

LoadedNexusData, RunNumber, ProposalNumber and DataPath are “domain types”.

The domain types are the nodes in the workflow graph, and they represent the inputs, intermediate results, or final results of the workflow.

Inputs, intermediate results, and final results#

A typical data reduction workflow has some “Inputs”, some “Intermediate results” and some “Final results”.

Sciline does not distinguish between those, but it is useful to make a loose distinction:

Inputs are typically:

The name of one or more NeXus files.
Parameters defining a region of interest (ROI),
- for example the wavelength range.
The number of histogram bins.

Intermediate results are typically:

List of events with associated coordinates, masks, and weights.
- “Coordinates” such as wavelength, scattering angle, etc.
Monitor wavelength histogram.
Various calibration factors that are computed or loaded from file.

Final results are typically:

Curve describing the scattering cross section \(S(Q)\) as a function of momentum transfer (SANS).
List of peaks and associated intensities (diffraction).
Etc.

Sciline graph example

How does Sciline know how to build the graph?#

Each domain type is unique, and given a list of all functions to be used in the workflow, Sciline can figure out how to connect the nodes in the graph.

Think of it like puzzle pieces that fit into each other:

Common errors#

In this section, we walk the reader through some of the most common errors that are encountered while working with sciline, and try to bring extra explanations to the error messages, which are not always simple to understand.

UnsatisfiedRequirement#

This is probably the most common error you will encounter; and it typically means you have forgotten to set a parameter on the workflow.

If we go back to our original workflow where we still had the Foldername and QBins red boxes

workflow = sl.Pipeline(
    # List the providers that make up the workflow.
    (load, to_wavelength, to_Q, to_histogram),
)

and we immediately try to compute the final histogram

workflow.compute(QHistogram)

---------------------------------------------------------------------------
UnsatisfiedRequirement                    Traceback (most recent call last)
Cell In[13], line 1
----> 1 workflow.compute(QHistogram)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/pipeline.py:191, in Pipeline.compute(self, tp, reporter, **kwargs)
    170 def compute(
    171     self,
    172     tp: type | Iterable[type] | "UnionType",  # noqa: UP037 (needed by Sphinx)
    173     reporter: Reporter | None = None,
    174     **kwargs: Any,
    175 ) -> Any:
    176     """
    177     Compute result for the given keys.
    178 
   (...)    189         Keyword arguments passed to the ``.get()`` method.
    190     """
--> 191     return self.get(tp, **kwargs).compute(reporter=reporter)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/pipeline.py:281, in Pipeline.get(self, keys, scheduler, handler, max_depth)
    278         info = f'{e} Requested node not in graph. Did you mean one of: {nodes}?'
    279     # Not raising `from e` because that includes noisy traceback of internals,
    280     # which are not relevant to the user.
--> 281     raise type(e)(f'{info}\n\n') from None
    282 return TaskGraph(
    283     graph=graph,
    284     targets=targets if multi else keys,  # type: ignore[arg-type]
    285     scheduler=scheduler,
    286 )

UnsatisfiedRequirement: Missing input node 'Foldername'. Affects requested targets (via providers given in parentheses):
1. Foldername → (__main__.load) → RawData → (__main__.to_wavelength) → WavelengthData → (__main__.to_Q) → Qdata → (__main__.to_histogram) → QHistogram

we are told that a requirement Foldername is missing to be able to compute the result.

Sciline stops at the first missing requirement.

A missing provider#

We build the same workflow, be here we forget to add one of the provider functions at build time:

workflow = sl.Pipeline(
    # Missing load provider!
    (to_wavelength, to_Q, to_histogram),
)

# Set the parameters as before
workflow[QBins] = sc.linspace("Q", 5.0e-3, 0.19, 201, unit="1/angstrom")
workflow[Foldername] = utils.fetch_data("3-mcstas/SANS_with_sample_many_neutrons")
workflow[CoordTransformGraph] = graph

# The graph looks different
workflow.visualize(graph_attr={"rankdir": "LR"})

../_images/d150cb7c09472b6e16096ae7f6d38c6ada7bbe81765471bfba0dd7e51500fe02.svg

workflow.compute(QHistogram)

---------------------------------------------------------------------------
UnsatisfiedRequirement                    Traceback (most recent call last)
Cell In[15], line 1
----> 1 workflow.compute(QHistogram)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/pipeline.py:191, in Pipeline.compute(self, tp, reporter, **kwargs)
    170 def compute(
    171     self,
    172     tp: type | Iterable[type] | "UnionType",  # noqa: UP037 (needed by Sphinx)
    173     reporter: Reporter | None = None,
    174     **kwargs: Any,
    175 ) -> Any:
    176     """
    177     Compute result for the given keys.
    178 
   (...)    189         Keyword arguments passed to the ``.get()`` method.
    190     """
--> 191     return self.get(tp, **kwargs).compute(reporter=reporter)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/pipeline.py:281, in Pipeline.get(self, keys, scheduler, handler, max_depth)
    278         info = f'{e} Requested node not in graph. Did you mean one of: {nodes}?'
    279     # Not raising `from e` because that includes noisy traceback of internals,
    280     # which are not relevant to the user.
--> 281     raise type(e)(f'{info}\n\n') from None
    282 return TaskGraph(
    283     graph=graph,
    284     targets=targets if multi else keys,  # type: ignore[arg-type]
    285     scheduler=scheduler,
    286 )

UnsatisfiedRequirement: Missing input node 'RawData'. Affects requested targets (via providers given in parentheses):
1. RawData → (__main__.to_wavelength) → WavelengthData → (__main__.to_Q) → Qdata → (__main__.to_histogram) → QHistogram

We are now in a split state where the error message is telling it is missing the RawData.

The workflow is supposed to get the RawData from loading the data in the folder, but because provider function linking Foldername to RawData is missing, the chain is broken and the Foldername is just an isolated box, not connected to any other parts of the graph.

Long error messages#

Sometimes, the error messages produced can appear dauntingly long.

A common example is when something goes wrong inside one of the provider functions.

The order in which the functions in the graph are called is managed/orchestrated by an external library called Dask, the calls happen deeply embedded in some scheduling mechanism that tries to optimize thread and memory use.

This means that when something fails inside one of the functions, the traceback will also contain a lot of Dask-specific information which users don’t actually need read or care about. Instead, one should scroll all the way to the bottom to read CoordError: Coordinate 'q' not found.

def bad_to_histogram(events: QData, qbins: QBins) -> QHistogram:
    """Histogram data in Q bins"""
    # Wrong coordinate name: it should be uppercase Q!
    return events.hist(q=qbins)


workflow = sl.Pipeline(
    (
        load,
        to_wavelength,
        to_Q,
        bad_to_histogram,
    ),
)

workflow[Foldername] = utils.fetch_data("3-mcstas/SANS_with_sample_many_neutrons")
workflow[CoordTransformGraph] = graph
workflow[QBins] = 200

# Uncomment the next line to see the exception
workflow.compute(QHistogram)

---------------------------------------------------------------------------
CoordError                                Traceback (most recent call last)
Cell In[16], line 21
     17 workflow[CoordTransformGraph] = graph
     18 workflow[QBins] = 200
     19 
     20 # Uncomment the next line to see the exception
---> 21 workflow.compute(QHistogram)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/pipeline.py:191, in Pipeline.compute(self, tp, reporter, **kwargs)
    170 def compute(
    171     self,
    172     tp: type | Iterable[type] | "UnionType",  # noqa: UP037 (needed by Sphinx)
    173     reporter: Reporter | None = None,
    174     **kwargs: Any,
    175 ) -> Any:
    176     """
    177     Compute result for the given keys.
    178 
   (...)    189         Keyword arguments passed to the ``.get()`` method.
    190     """
--> 191     return self.get(tp, **kwargs).compute(reporter=reporter)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/task_graph.py:122, in TaskGraph.compute(self, targets, reporter)
    120     return dict(zip(targets, results, strict=True))
    121 else:
--> 122     return self._scheduler.get(self._graph, [targets], reporter=reporter)[0]

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/scheduler.py:64, in NaiveScheduler.get(self, graph, keys, reporter)
     62 with reporter.run_computation(graph.values()):
     63     for t in tasks:
---> 64         results[t] = reporter.call_provider_with_reporting(graph[t], results)
     65 return tuple(results[key] for key in keys)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/reporter.py:398, in NullReporter.call_provider_with_reporting(self, provider, values)
    396 """Call a provider and report its progress with this reporter."""
    397 # Override base method to avoid overhead.
--> 398 return provider.call(values)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/sciline/_provider.py:144, in Provider.call(self, values)
    142 def call(self, values: dict[Hashable, Any]) -> Any:
    143     """Call the provider with arguments extracted from ``values``."""
--> 144     return self._func(
    145         *(values[arg] for arg in self._arg_spec.args),
    146         **{key: values[arg] for key, arg in self._arg_spec.kwargs},
    147     )

Cell In[16], line 4, in bad_to_histogram(events, qbins)
      1 def bad_to_histogram(events: QData, qbins: QBins) -> QHistogram:
      2     """Histogram data in Q bins"""
      3     # Wrong coordinate name: it should be uppercase Q!
----> 4     return events.hist(q=qbins)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/data_group.py:755, in data_group_overload.<locals>.impl(data, *args, **kwargs)
    753 if isinstance(data, DataGroup):
    754     return data.apply(impl, *args, **kwargs)  # type: ignore[arg-type]
--> 755 return func(data, *args, **kwargs)

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/binning.py:584, in hist(x, arg_dict, dim, **kwargs)
    580 if isinstance(x, DataGroup):
    581     # Only to make mypy happy because we have `DataGroup` in annotation of `x`
    582     # so that Sphinx shows it.
    583     raise TypeError("Internal error: input should not be a DataGroup")
--> 584 edges = _make_edges(x, arg_dict, kwargs)
    585 erase = _find_replaced_dims(x, dims=edges, dim=dim)
    586 if isinstance(x, Variable) and len(edges) != 1:

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/binning.py:385, in _make_edges(x, arg_dict, kwargs)
    379 def _make_edges(
    380     x: Variable | DataArray | Dataset,
    381     arg_dict: IntoStrDict[SupportsIndex | Variable] | None,
    382     kwargs: Mapping[str, SupportsIndex | Variable],
    383 ) -> dict[str, Variable]:
    384     return {
--> 385         name: _parse_coords_arg(x, name, arg)
    386         for name, arg in combine_dict_args(arg_dict, kwargs).items()
    387     }

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/binning.py:348, in _parse_coords_arg(x, name, arg)
    346 if isinstance(arg, Variable) and name in arg.dims:
    347     return arg
--> 348 coord = _get_coord(x, name)
    349 start = coord.nanmin()
    350 if (
    351     not isinstance(x, Variable)
    352     and (name in x.coords)
    353     and x.coords.is_edges(name, name)
    354 ):

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/binning.py:326, in _get_coord(x, name)
    324     event_coord = x.bins.coords.get(name) if x.is_binned else None
    325     coord = x.coords.get(name, event_coord)
--> 326 _require_coord(name, coord)
    327 return coord

File ~/work/dmsc-school/dmsc-school/.pixi/envs/default/lib/python3.12/site-packages/scipp/core/binning.py:307, in _require_coord(name, coord)
    305 def _require_coord(name: str, coord: object) -> None:
    306     if coord is None:
--> 307         raise CoordError(f"Coordinate '{name}' not found.")

CoordError: Coordinate 'q' not found.

Generic domain types#

Sometimes we want to replicate parts of a workflow and apply it to a different input.

A typical case is when we have a sample measurement and want to correct it by a background measurement.

In that case many of the processing steps are identical, but ultimately we want to subtract the background measurement from the sample measurement.

Generic domain types lets us define domain types that represent “Y of the X” such as:

Filename[Background]: Filename of the background run.
QHistogram[Sample]: The Q-histogram of the sample run.
QHistogram[Background]: The Q-histogram of the background run.
etc

from typing import TypeVar


# Define concrete RunType values we will use.
Sample = NewType("Sample", int)
Background = NewType("Background", int)

# Define generic domain types
RunType = TypeVar("RunType", Sample, Background)

# Domain types

# We use sciline.Scope to make Filename a "generic" domain type that depends on RunType.
class Foldername(sl.Scope[RunType, str], str): ...

class RawData(sl.Scope[RunType, sc.DataArray], sc.DataArray): ...

class WavelengthData(sl.Scope[RunType, sc.DataArray], sc.DataArray): ...

class QData(sl.Scope[RunType, sc.DataArray], sc.DataArray): ...

class QHistogram(sl.Scope[RunType, sc.DataArray], sc.DataArray): ...

FinalHistogram = NewType("FinalHistogram", sc.DataArray)
"""Data histogrammed in Q bins where background has been subtracted"""


# Note that the QBins and coordinate transform graph are the same for
# Sample and Background runs, so there is no need to make them into
# generic types.

CoordTransformGraph = NewType("CoordTransformGraph", dict)

QBins = NewType("QBins", sc.Variable)



def load(folder: Foldername[RunType]) -> RawData[RunType]:
    """Load raw data from file"""
    return utils.load_sans(folder)


def to_wavelength(data: RawData[RunType], graph: CoordTransformGraph) -> WavelengthData[RunType]:
    """Compute wavelength for events"""
    return data.transform_coords("wavelength", graph=graph)


def to_Q(data: WavelengthData[RunType], graph: CoordTransformGraph) -> QData[RunType]:
    """Compute Q for events"""
    return data.transform_coords("Q", graph=graph)


def to_histogram(events: QData[RunType], qbins: QBins) -> QHistogram[RunType]:
    """Histogram data in Q bins"""
    return events.hist(Q=qbins)

# A new function that will subtract background from sample run.
# Note here that instead of [RunType], we have [Sample] and [Background]
def subtract_background(
    sample_data: QHistogram[Sample],
    background_data:QHistogram[Background],
) -> FinalHistogram:
    """Subtract background signal"""
    return sample_data - background_data


workflow = sl.Pipeline(
    # List the providers that make up the workflow.
    (load, to_wavelength, to_Q, to_histogram, subtract_background),
)

workflow.visualize(graph_attr={"rankdir": "LR"})

../_images/fd8e5be8951de4310700d3550719d2f676bb8187c69be1d35ef0f34c87a4fee4.svg

To set the parameter, we must not forget to also use the [Sample] and [Background] specifiers inside the square brackets:

workflow[Foldername[Sample]] = utils.fetch_data("3-mcstas/SANS_with_sample_many_neutrons")
workflow[Foldername[Background]] = utils.fetch_data("3-mcstas/SANS_without_sample_many_neutrons")
workflow[QBins] = sc.linspace("Q", 5.0e-3, 0.19, 201, unit="1/angstrom")
workflow[CoordTransformGraph] = graph

And we can finally compute the final result:

To compute an intermediate result, we also need to use the [Sample] and [Background] specifiers:

How will Sciline be used at ESS?#

Most instruments will have one or more associated Sciline workflows.

The workflows will be the basic interface to the data reduction software.

On top of that interface we can build simpler but less flexible interfaces.

But that will take time.
In the early days after HC the interface to the data reduction will be mainly in the form of Sciline workflows.

What do I need to know?#

How to figure out what quantity to compute with the workflow.
- Look at the workflow graph and read on the technique package documentation page.
How to figure out what parameters are needed to compute the target quantity.
- Error messages tell you what is missing, or you can look at the workflow graph.
How to set parameters on the workflow.
How to compute the desired quantity.
How to read and understand common error messages.

Introduction to Sciline

Contents

Introduction to Sciline#

Data processing workflow management tool.

scipp.github.io/sciline/

Terminology#

Inputs, intermediate results, and final results#

How does Sciline know how to build the graph?#

Example#

Creating a pipeline#

Setting parameters#

Computing quantities#

Intermediate results#

Computing multiple results#