Database Ingestion and Live-prep

[Sceki]

@dead-water @will-fawcett porting here the issue. I also added an item about testing w.r.t. existing db & soho/omni/goes support

Infrastructure & code:

• Set up InfluxDB
• Check that all live sources are in same format as archived static sources
• If not, consider translation such that there is a general format for all data sources
• Prepare scripts for live data standardisation post ingestion
• do same for soho
• do same for omni
• do same for goes
• test that influxDB data are identical to the csv initially prepared in the 'static' approach, at random dates

For ML readiness

• All static datasets uploaded
• Cloud functions ready for Terraform deployment

[Sceki]

Pasting here from @dead-water

# Data prep
import pandas as pd
champ = pd.read_csv("/shared/us-hl-therm-landing/champ_data.csv")
champ["Time (UTC)"] = pd.to_datetime(champ["Time (UTC)"])
champ = champ.reset_index().set_index("Time (UTC)", drop=True)
champ = champ.drop(columns=["index", "Unnamed: 0"])

# Database connection
from influxdb_client import InfluxDBClient

url = ''
token = ''
org = 'FDL'
bucket = 'us-hl-therm-landing'

# Writing
with InfluxDBClient(url=url, org=org, token=token) as client:
    print(client.health())

    with client.write_api() as write_api:
        write_api.write("hl-therm-influxdb", "FDL", record=champ, data_frame_measurement_name='champ',
                                data_frame_field_columns=champ.columns)
# Reading/Checking
with InfluxDBClient(url=url, org=org, token=token) as client:
    print(client.health())

    # with client.query_api() as query_api:
    query_api = client.query_api()
    tables = query_api.query('from(bucket: "hl-therm-influxdb") |> range(start: -100y)')

    for table in tables:
        for record in table.records:
            print(str(record["_time"]) + " - " + record.get_measurement()
                + " " + record.get_field() + "=" + str(record.get_value()))