[V3] Directory Loading and Begin Implementing Lazy-Loading

[project-laguardia]

Problem:

As it stands, Dasel does not lazy load to any extent, which can cause performance issues for large datasets.

Solution (Getter-Setters):

Dasel already has groundwork implemented for dynamically-loading properties (implemented for orderedmaps) in model.Value:

• dasel/model/value_map.go

  Lines 85 to 127 in 5ac2dab

  	func (v *Value) GetMapKey(key string) (*Value, error) {
  	switch {
  	case v.isDencodingMap():
  	m, err := v.dencodingMapValue()
  	if err != nil {
  	return nil, fmt.Errorf("error getting map: %w", err)
  	}
  	val, ok := m.Get(key)
  	if !ok {
  	return nil, MapKeyNotFound{Key: key}
  	}
  	res := NewValue(val)
  	res.setFn = func(newValue *Value) error {
  	m.Set(key, newValue.Value.Interface())
  	return nil
  	}
  	return res, nil
  	case v.isStandardMap():
  	unpacked, err := v.UnpackUntilKind(reflect.Map)
  	if err != nil {
  	return nil, fmt.Errorf("error unpacking value: %w", err)
  	}
  	i := unpacked.Value.MapIndex(reflect.ValueOf(key))
  	if !i.IsValid() {
  	return nil, MapKeyNotFound{Key: key}
  	}
  	res := NewValue(i)
  	res.setFn = func(newValue *Value) error {
  	mapRv, err := v.UnpackUntilKind(reflect.Map)
  	if err != nil {
  	return fmt.Errorf("error unpacking value: %w", err)
  	}
  	mapRv.Value.SetMapIndex(reflect.ValueOf(key), newValue.Value)
  	return nil
  	}
  	return res, nil
  	default:
  	return nil, ErrUnexpectedType{
  	Expected: TypeMap,
  	Actual: v.Type(),
  	}
  	}
  	}

• dasel/model/value_map.go

  Lines 41 to 60 in 5ac2dab

  	func (v *Value) SetMapKey(key string, value *Value) error {
  	switch {
  	case v.isDencodingMap():
  	m, err := v.dencodingMapValue()
  	if err != nil {
  	return fmt.Errorf("error getting map: %w", err)
  	}
  	m.Set(key, value.Value.Interface())
  	return nil
  	case v.isStandardMap():
  	unpacked, err := v.UnpackUntilKind(reflect.Map)
  	if err != nil {
  	return fmt.Errorf("error unpacking value: %w", err)
  	}
  	unpacked.Value.SetMapIndex(reflect.ValueOf(key), value.Value)
  	return nil
  	default:
  	return fmt.Errorf("value is not a map")
  	}
  	}

Using this existing getter-setter structure, you could add support for lazily-loaded types that only load into memory on demand.

Use Case: Directory Loading

A simplistic first use-case would be directory loading. When parsing the directory, you first only load the directory tree into memory as a map. Then when leaves (files) are accessed, you load those on demand. This would give you a poor man's lazy-loading method for large datasets by allowing end users to segment their files.

[project-laguardia]

A few special considerations for Directory Loading:

• Directory tree file and folder names would have to be constrained by the v3 AST (they may not be able to contain special chars)
• File format loading may have to be dynamic.
  • A solution that loads files by using basename as the "property" and extension for format detection is likely desired (i.e. path/to/file.yaml would be loaded as path.to.file where file is a yaml document.
    • The primary problem here would be if 2 files share the same basename (i.e. path/to/file.yaml and path/to/file.json)
      • Maybe a solution here would be to allow loading via [] operator as well, if you need to specify the full file name (i.e. path.to["file.yaml"] and path.to["file.json"])
        • Then have a directory reading flag for specifying preferred filetype for basename-accessors (i.e. if flag is set to "yaml," path.to["file.yaml"] and path.to.file both resolve to the same data, but path.to["file.json"] resolves to something different)
• Directory tree writing would be a complex issue. For now, aggregating the query result into a single (and separate) stdout would be an acceptable first step, but a solution for writing back to the tree may also be desired at a later point.
  • Perhaps one solution to directory writing would be to track which nodes stemming away from the root directory map come from the file system and which ones come from actual data, then writing data-sourced information to the pertaining file (or new file if a leaf was added to the tree).

[project-laguardia]

Perhaps you could add a third check called model.Value.isDynamicMap() or model.Value.isLazyMap()

[TomWright]

I doubt dasel v3 will introduce directory based lazy loading - rather file based lazy loading where the file contents are gradually streamed into memory as they are queried. Most use-cases only wish to operate on a single file, even stdin so directory based loading is unnecessary.

A readFile (or similar) function is probably sufficient, assuming the file is also lazy loaded/streamed for most usecases, e.g.

dasel '$x = readFile(x.json);
$x.foo.bar'

[TomWright]

To clarify I see lazy loading/file streaming as a separate feature to directly loading

[project-laguardia]

No, I get it. Directory loading is just a poor man's lazyloading