TypeError: '>' not supported between instances of 'NoneType' and 'int' when trying to use fio-plot to plot a 2D line chart after running io test by bench-fio

[enkka]

I got a pytyhon error when trying to use fio-plot to plot a 2D line chart after running io test by bench-fio

Traceback (most recent call last):
File "/usr/local/bin/fio-plot", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/fio_plot/init.py", line 40, in main
data = routing_dict[graphtype]"get_data"
File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/getdata.py", line 33, in get_log_data
logfiles = logdata.filterLogFiles(settings, benchmarkfiles)
File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/dataimport.py", line 111, in filterLogFiles
data["directory"] = return_folder_name(item["filename"], settings, True)
File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/dataimport.py", line 59, in return_folder_name
if child > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

here is bench-fio ini config

[benchfio]
type=device
direct=1
engine=libaio
destructive=True
parallel=True
mode=write
runtime=60
block_size=1m
output=./20231030-083144
iodepth=8
numjobs=1
target=/dev/sda

fio-plot ini config

[graphtype]
graphtype = loggraph

[settings]
output_filename = 20231030-083144.png
title = write 1m for 60s test result
source = 20231030-083144
rw = write
iodepth = 8
numjobs = 1
type = bw
input_directory = /root/20231030-083144/SN0077_sda/1m/
filter = read,write
maxdepth = 64
maxjobs = 64
dpi = 300
percentile = 99.99
max_z =
max_lat =
max_iops =
max_bw =
moving_average = 

[layout]
title_fontsize = 16
subtitle_fontsize = 10
source_fontsize = 6
credit_fontsize = 8
table_fontsize = 10
xlabel_depth = 
xlabel_parent = 
xlabel_segment_size = 
line_width = 
grou_bars = 
show_cpu = False
show_ss = False
table_lines = False
disable_grid = False
enable_markers = False
disable_fio_version = False
colors = 

[enkka]

20231030-083144.zip
attached bench-fio test result

[louwrentius]

Thanks for your report and the data. I will take a look into it

[louwrentius]

I'm able to exactly reproduce the issue, so that's a start.

If I create a manual graph (not using ini)

../projects/fio-plot/bin/fio-plot -i array/sda/1m/ -T "Test" -g -t bw -r write -d 8 -n 1

I know there is an issue with merging default settings and the ini settings, which is more complex than I thought. Needs more work.

[louwrentius]

I have fixed the issue in a new release. Let me know if you encounter any issues.

[enkka]

Hi,

Thanks for update.

When I added

xlabel_depth = 0
xlabel_parent = 0
xlabel_segment_size = 0

to [settings] segment in fio-plot ini file, I got this error when trying to run fio-plot with ini file.

Traceback (most recent call last):
  File "/usr/local/bin/fio-plot", line 33, in <module>
    sys.exit(load_entry_point('fio-plot==1.1.8', 'console_scripts', 'fio-plot')())
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/__init__.py", line 41, in main
    routing_dict[graphtype]["function"](settings, data)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/graph2d.py", line 25, in chart_2d_log_data
    data = supporting.process_dataset(settings, dataset)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/supporting.py", line 261, in process_dataset
    if settings["draw_total"] and len(settings["filter"]) == 2:
KeyError: 'draw_total'

after checking supporting.py, I found the sentence

if settings["draw_total"] and len(settings["filter"]) == 2:

but in my ini file the draw_total is not available.

If I added draw_total = False to to [settings] segment in fio-plot ini file, the error changes to following.

Traceback (most recent call last):
  File "/usr/local/bin/fio-plot", line 33, in <module>
    sys.exit(load_entry_point('fio-plot==1.1.8', 'console_scripts', 'fio-plot')())
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/__init__.py", line 41, in main
    routing_dict[graphtype]["function"](settings, data)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/graph2d.py", line 25, in chart_2d_log_data
    data = supporting.process_dataset(settings, dataset)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/supporting.py", line 262, in process_dataset
    readdata = record["read"]["yvalues"]
TypeError: list indices must be integers or slices, not str

Since the io test only run single seq write test as before, there is no read data can be get by script.

Could you please take a look at the "draw_total" coupling with default setting of "filter = read,write" ? Thank you.

[louwrentius]

Thanks for your report and sorry to hear that you encountered these issues. I'll take a look!

@enkka xlabel_segment_size = 0 results in a graph with no labels as it's equivalent to string length = 0. What are you trying to achieve?

The draw-total parameter is to sum both read+write data, if read data is not available, the option should not work and I'll have to add some checks on that.

I've run your ini file with your settings and I notice I still didn't fix the ini parsing as the xlabel settings in the ini aren't honoured and the defaults still apply so I'll have to fix that first.

to [settings] segment in fio-plot ini file, I got this error when trying to run fio-plot with ini file.

@enkka you have to add the xlabel settings to the layout section, not the settings section

[louwrentius]

I did upload a new version of fio-plot to fix some other issues. (1.1.9)..

[enkka]

Thanks for your report and sorry to hear that you encountered these issues. I'll take a look!

@enkka xlabel_segment_size = 0 results in a graph with no labels as it's equivalent to string length = 0. What are you trying to achieve?

The draw-total parameter is to sum both read+write data, if read data is not available, the option should not work and I'll have to add some checks on that.

I've run your ini file with your settings and I notice I still didn't fix the ini parsing as the xlabel settings in the ini aren't honoured and the defaults still apply so I'll have to fix that first.

to [settings] segment in fio-plot ini file, I got this error when trying to run fio-plot with ini file.

@enkka you have to add the xlabel settings to the layout section, not the settings section

Thanks @louwrentius .

1. xlabel_segment_size = 0 results in a graph with no labels as it's equivalent to string length = 0. What are you trying to achieve?
   A: it is just for test, since I found I cannot leave it blank. or else I will got this error

Traceback (most recent call last):
  File "/usr/local/bin/fio-plot", line 33, in <module>
    sys.exit(load_entry_point('fio-plot==1.1.9', 'console_scripts', 'fio-plot')())
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/__init__.py", line 40, in main
    data = routing_dict[graphtype]["get_data"](settings)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/getdata.py", line 33, in get_log_data
    logfiles = logdata.filterLogFiles(settings, benchmarkfiles)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/dataimport.py", line 111, in filterLogFiles
    data["directory"] = return_folder_name(item["filename"], settings, True)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/dataimport.py", line 59, in return_folder_name
    if child > 0:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

2. @enkka you have to add the xlabel settings to the layout section, not the settings section
   Yes I will refer to the fio-plot.ini file under fio-plot-1.1.9\fio_plot\templates folder.

3. 1 more issue I want to feedback: I found those optional arguments seems no default value, i.e. leave colors to blank by adding 'colors = ' to layout segment in ini file, I will get this info in return.
   Color is not a known color. Please check the spelling.
   Similar issue as arguments truncate_xaxis, subtitle, include_hosts, exclude_hosts. I have to add those into ini files or else I got error messages.

Traceback (most recent call last):
  File "/usr/local/bin/fio-plot", line 33, in <module>
    sys.exit(load_entry_point('fio-plot==1.1.9', 'console_scripts', 'fio-plot')())
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/__init__.py", line 41, in main
    routing_dict[graphtype]["function"](settings, data)
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/graph2d.py", line 109, in chart_2d_log_data
    if supporting.filter_hosts(settings, item):
  File "/usr/local/lib/python3.10/dist-packages/fio_plot/fiolib/supporting.py", line 497, in filter_hosts
    if settings["include_hosts"]:
KeyError: 'include_hosts'

[enkka]

If those optional arguments like "truncate_xaxis, subtitle" are required, it is fine, we can add it to the template file.
I also suggest adding those into fio-plot-1.1.9\fio_plot\templates\fio-plot.ini as well.

[louwrentius]

I will try and reproduce these issues, but the latest version(s) are specifically meant to fix exactly these issues.