Replies: 7 comments 15 replies
-
|
@bl-young, there's been a discussion on our end whether we need to 'bulk up' the flow metadata in the baseline's JSON-LD. Currently, I'm using fedelemflowlist v.1.2.2 to map elementary flows to get their UUIDs; however, I'm seeing that the CAS number and the FEDEFL version numbers are not carried over. The Commons seems to have version 1.0 available. Any recommendations on paths forward to meet requirements and utilize exiting tools? |
Beta Was this translation helpful? Give feedback.
-
|
I was not aware of that! Thanks, Ben. I'll give it a try. |
Beta Was this translation helpful? Give feedback.
-
|
Hangup might be a strong word. I think that all the unit process metadata is there to satisfy FLCAC requirements...or at least they were satisfied at the time. Along the topic of flows though, we do create a number of product and waste flows (generated by RCRAInfo) that don't fit within the elementary flows list. There are no metadata associated with these, and unfortunately, I don't know what flow-level metadata requirements there are in the commons. We should also probably move the waste flows from "Elementary flows/emission/waste" to "Technosphere Flows" or make a new "Waste Flows" category. Guess I'll call out @bl-young again here - has there been a standard set somewhere on waste flows as provided by StEWI? |
Beta Was this translation helpful? Give feedback.
-
|
Yes RCRA... I believe we have these from USEEIO as well. When we turned that into JSON this is how it manifests: Is that the best solution? I may need to take that back and think on it in terms of the specific categories/contexts for those flows. I'm open to adjusting the USEEIO repo for those flows if that makes sense and you all have a better solution. I do recommend using the UUID function in esupy (here] for consistent UUID generation. It would be good if the same RCRA flows always had the same UUID and contexts across repos. I also seem to recall the existing baseline having some additional elementary flows that arent in the FEDEFL, maybe they were some water flows? Happy to set something up to talk through this in more detail and/or review what you have so far if that's useful. I'll also add I have been working on creating a schema for Sources through esupy (USEPA/esupy#51) if thats valuable for you. |
Beta Was this translation helpful? Give feedback.
-
|
@dt-woods in FLCAC-admin/LCIAformatter#103 I've been wanting to use the existing function in fedelemflowlist to write flows in lciafmt to make them more consistent. I started that with FLCAC-admin/LCIAformatter@36a37c4 and FLCAC-admin/fedelemflowlist@7ed509a. I realized that the way things are currently set up, it works best to pass the |
Beta Was this translation helpful? Give feedback.
-
|
I'm pretty sure our UUID generation is the same as esupy's, but good to know that we would could (maybe even should) replace it with an import to that method. It assumes that the args we send the method are consistent. |
Beta Was this translation helpful? Give feedback.
-
|
Here's my proposed solution, which happens within olca_jsonld_writer.py: def _flow(dict_d, flowprop, dict_s):
"""Generate a reference to a flow object.
Called by :func:`_exchange`.
Parameters
----------
dict_d : dict
Flow data dictionary.
flowprop : olca_schema.FlowProperty or olca_schema.Ref
FlowProperty or Ref to a FlowProperty object.
dict_s : dict
Dictionary with olca_schema root entities.
Returns
-------
tuple
olca_schema.Ref : Reference to a flow object.
dict : Updated dictionary with olca_schema root entities.
"""
if not isinstance(dict_d, dict):
logging.warning("No flow data received!")
return (None, dict_s)
uid = _val(dict_d, 'id', '@id')
name = _val(dict_d, 'name')
category_path = _val(dict_d, 'category', default='')
is_waste = "waste" in category_path.lower()
# HOTFIX: remove technosphere/3rd party flow check;
# it duplicates every waste flow in the JSON-LD [2023-12-05; TWD]
# Check for flow existence
if uid in dict_s['Flow']['ids']:
idx = dict_s['Flow']['ids'].index(uid)
flow = dict_s['Flow']['objs'][idx]
logging.debug("Found previous flow, '%s'" % flow.name)
else:
logging.debug("Creating new flow for, '%s' (%s)" % (name, uid))
if _uid_is_valid(uid, 3) or _uid_is_valid(uid, 4):
# Keep the good UUID
pass
else:
# Generate new v3 ID based on standard format
logging.debug("Generating new UUID for flow, '%s'" % name)
uid = _uid(o.ModelType.FLOW, category_path, name)
# Correct the default flow type for waste flows.
def_type = "ELEMENTARY_FLOW"
if is_waste:
dict_d['flowType'] = "WASTE_FLOW"
def_type = "WASTE_FLOW"
f_type = _flow_type(_val(dict_d, 'flowType', default=def_type))
flow = o.new_flow(name, f_type, flowprop)
# Get FEDEFL version number and CAS number
fed_version = _get_fedefl_version()
fed_cas, fed_form, fed_syn = _get_fedefl_meta(uid)
# Hotfix: Add missing flow metadata.
# https://github.com/USEPA/ElectricityLCI/discussions/239
flow.id = uid
flow.category = category_path
flow.description = "Based on FEDELEMFLOW version %s" % fed_version
flow.cas = fed_cas
flow.formula = fed_form
flow.synonyms = fed_syn
# Update master list
dict_s['Flow']['ids'].append(uid)
dict_s['Flow']['objs'].append(flow)
return (flow.to_ref(), dict_s)
def _get_fedefl_meta(flow_id):
"""Query the FEDEFL for flow metadata (i.e., CAS No, formula, and synonyms)
Parameters
----------
flow_id : str
Flow universally unique identifier (mapped to FEDEFL).
Returns
-------
tuple
A tuple of length three:
- str: CAS No
- str: Formula
- str: Synonyms
"""
# Initialize return values
f_cas = ""
f_form = ""
f_syn = ""
# Read the FEDEFL into memory
flowlist = fedelemflowlist.get_flows(False)
# Query the flow ID and count rows that match
tmp_df = flowlist.query("`Flow UUID` == '%s'" % flow_id)
num_matches = len(tmp_df)
# Query results for metadata.
if num_matches == 0:
logging.info("Failed to find FEDEFL flow (%s)!" % flow_id)
elif num_matches == 1:
# Convert data frame to series
tmp_df = tmp_df.iloc[0]
f_cas = tmp_df['CAS No']
f_form = tmp_df['Formula']
f_syn = tmp_df['Synonyms']
else:
logging.warning(
"Found %d matches for FEDEFL flow (%s); using first instance." % (
num_matches, flow_id)
)
tmp_df = tmp_df.iloc[0]
f_cas = tmp_df['CAS No']
f_form = tmp_df['Formula']
f_syn = tmp_df['Synonyms']
return (f_cas, f_form, f_syn)
def _get_fedefl_version():
"""Return the version string of Federal Elementary Flow List
Currently hard-coded to return version 1.0.0 as defined in USEPA's
LCIAfmt. See https://github.com/USEPA/fedelemflowlist/issues/199.
Returns
-------
str
Version string (e.g., '1.0.0' or 'n/a' if not available)
"""
try:
# Wishy thinking
return fedelemflowlist.__version__
except AttributeError:
return "1.0.0" |
Beta Was this translation helpful? Give feedback.
-
|
for Though really, I'll caution that there is a distinction between the version of the package and the version of the flowlist (you could have the latest version of fedelemflowlist but you might have an older version of the FlowList data file (parquet) stored locally which could get used). |
Beta Was this translation helpful? Give feedback.
-
|
Good to know! Yeah, I want the version of the flow list, not the Python package. I'll incorporate the code you sent. Thanks, Ben! |
Beta Was this translation helpful? Give feedback.
-
|
What I'm seeing as a means to getting the file version is the following: from esupy.processed_data_mgmt import find_file
import fedelemflowlist
def _get_fedefl_version():
"""Return the version string of local Federal Elementary Flow List
Returns
-------
str
Version string. Defaults to '1.0.0' if methods fail.
"""
version = None
try:
# Adapted from:
# https://github.com/USEPA/ElectricityLCI/discussions/239
meta = fedelemflowlist.globals.set_metadata()
fpath = fedelemflowlist.globals.fedefl_path
# From esupy's documentation: returns the most recent version of the
# file requested---so does not have to match ``meta.tool_version``.
f = find_file(meta, fpath)
if f:
# The parquet file name is based on ``set_metadata`` function
# in fedelemflowlist's globals.py.
r = re.compile(".*_v(\d+\.\d+\.\d+)_.*")
m = r.match(f.name)
if m:
version = m.group(1)
if version is None:
version = fedelemflowlist.globals.flow_list_specs['list_version']
except AttributeError:
version = "1.0.0"
return version |
Beta Was this translation helpful? Give feedback.
-
|
Moving forward we will be distinguishing the package version from the list version (see FLCAC-admin/fedelemflowlist#205). Did you end up implementing this function above as I think it could be useful to move into fedelemflowlist itself. |
Beta Was this translation helpful? Give feedback.
-
|
Running |
Beta Was this translation helpful? Give feedback.
-
|
see my proposed solution in KeyLogicLCA#12 |
Beta Was this translation helpful? Give feedback.
-
|
Despite the slowness, it gets the job done. |
Beta Was this translation helpful? Give feedback.
-
|
I think it would be a benefit to move this step to the post processing phase. |
Beta Was this translation helpful? Give feedback.
-
|
How do I know what version of the parquet file I have?
…On Thu, Aug 29, 2024, 14:12 Ben Young (ERG) ***@***.***> wrote:
for _get_fedefl_version(), could you instead use:
fedelemflowlist.globals.flow_list_specs['list_version']
Though really, I'll caution that there is a distinction between the
version of the package and the version of the flowlist (you could have the
latest version of fedelemflowlist but you might have an older version of
the FlowList data file (parquet) stored locally which could get used).
—
Reply to this email directly, view it on GitHub
<#239 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHCFB5MHPOX5QAUBBYQZWF3ZT5P7XAVCNFSM6AAAAABGTK5GK6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBZGA2DGNI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
there should be a metadata file (json) that is downloaded as well that indicates that. Also the name of the file should include the version number. |
Beta Was this translation helpful? Give feedback.
-
Process Metadata YAMLHere's a little work on figuring out what is assigned where. The metadata keys:
|
Beta Was this translation helpful? Give feedback.
-
|
Several of these issues are solved thanks to correcting the |
Beta Was this translation helpful? Give feedback.
-
|
Notably, the YAML documentation is parsed using In upstream_dict.py, the fuel categories are used to find the appropriate entry in process_metadata.yml (by appending "_upstream" to the fuel category name). |
Beta Was this translation helpful? Give feedback.
-
|
Found the potential problem with missing fuel categories in VALID_FUEL_CATS here (no biomass and no hydro): |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
What are they?
Beta Was this translation helpful? Give feedback.
All reactions