Worked Examples

Getting Started: Generators and Scheduled Loads

A great place to start exploring the AEMO datasets is the list of Generators and Scheduled Loads, this data is from AEMO's list of registrations. Access it through NEMOSIS by following the steps below, where a predefined session is used.

• Follow this link and download the latest release of the graphical user interface as an executable file.
• Download the session file for this example here
• Open the NEMOSIS.exe file, windows will not recognise this file, so you need to inform it that it is safe to run, click through any warnings
• Click the load session button and then select the example_zero.pkl file you just downloaded
• You will also need to create two new folders on you computer, one for the raw data cache and one to save the results in. Then provide the paths to these folders to NEMOSIS in the text entry boxes provided.
• Start and end times are not given for the Generators and Scheduled Loads table as this is a static table and its contents do not change with respect to time
• Now you should be ready to run the query, click the 'Run queries' button. Any time you run a query that requires new data to be downloaded, it may take some time while NEMOSIS creates and formats the local data cache.

Now the data should be ready for you to view. Explore the generator set and try using generators' DUIDs to filter other queries like those outlined below.

Filtered Query of Dispatch Targets

This example outlines how to construct a query for dispatch targets of a single dispatch unit (Hallett 2 Wind Farm). The steps below describe how to create your own query from scratch.

• The query is given a name, this will be used when NEMOSIS saves the results as a csv
• A start and end time are given
• The DISPATCH_LOAD table is selected as this contains the dispatch targets data
• The columns 'SETTLEMENTDATE', 'DUID' and 'INTERVENTION' are automatically selected by NEMOSIS as they form the primary key for this table
• Additionally select the column 'TOTALCLEARED' to get the dispatch targets for each interval
• The filter for Hallett 2 Wind Farm is created by entering its DUID (HALLWF2) into the filter entry box, pressing enter and then selecting it from the list below
• The filter for the intervention flag is automatically opened by NEMOSIS, but will be ignored unless values are added and selected.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.

SCADA Data for Solar Generators

This example outlines how to query for SCADA data and generator fuel types. Additionally, it demonstrates how to merge datasets.

• In the AEMO database generator information and unit SCADA data are stored separately, therefore in order create a dataset of SCADA data for just solar generators we need to query two datasets
• First all the SCADA data is pulled unfiltered, then all the fuel type data is pulled from the Generators and Scheduled Loads table but is filtered to just return 'Solar' generators
• When an inner join is used to merge the datasets on the 'DUID' column all non solar generators are excluded from the result as they are not in the right hand table
• In this example, query names not only provide file names for saving data, but are also referenced when merging data.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.

Custom tables | 4 second Data | Crashing the Query

NEMOSIS is also equipped to host algorithms for producing custom data tables that synthesise publicly available AEMO data into something new and interesting. One example of this is the table FCAS_4s_SCADA_MAP. AEMO records 4 second load, generation and interconnector data for the purposes of the FCAS Causer Pays calculations. However, the names in this dataset are from the Energy Management System (EMS) while most other tables use names from the Market Management System (MMS). AEMO does not provide publicly available mapping between these names. This is the job of the FCAS_4s_SCADA_MAP table, linking 4 s generation data with other data like generator fuel types. This opens interesting possibilities when unpicking 'sticky' ideas like generator 'reliability'. However, 4 s data for all generators in the NEM is extremely memory intensive! If you ask for too much data it will happily crash your query with a memory error and not return any data. This particularly applies to the algorithm that creates the FCAS_4s_SCADA_MAP table.

Note the algorithm behind the mapping table is still under development.

Example: 4 s Data for Hallett 2 Wind Farm

Here we are going to pull the 4 s data for Hallett 2 Wind Farm from the same time period we looked at previously, its good to choose this time period because we can be confident that the wind farm is on, and therefore that the matching algorithm will work. This is because the algorithm works by comparing MMS SCADA data to the FCAS 4 s SCADA.

• First create the query for the FCAS_4s_SCADA_MAP, filter this with the MARKETNAME (DUID) 'HALLWF2' and the start and end time shown below, here we contract the time window significantly as the matching algorithm has higher memory requirements then just pulling the 4 s data. Also we won't select the 'ERROR' column as this is mostly for monitoring the accuracy of the matches
• Then create the query to pull the raw FCAS_4_SECOND table, the memory requirements for just pulling the data are lower, so we can broaden the time window here
• Lastly use an inner join of the two previous datasets to leave us just with the 4 s data for Hallett 2 Wind Farm.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.