Skip to content

Add AIO and Direct IO support for IOR phases #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jxiong wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add AIO and Direct IO support for IOR phases #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions prepare.sh
View file
Open in desktop
Copy link
Copy Markdown
Contributor

@JulianKunkel JulianKunkel Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't accept these changes, anyone can modify it. We will integrate some change to clarify that you are allowed to change it.

Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ function get_schema_tools {
function build_ior {
pushd "$BUILD"/ior
./bootstrap
# Add here extra flags
./configure --prefix="$INSTALL_DIR"
# Add here extra flags (e.g., --with-aio for asynchronous I/O support)
./configure --prefix="$INSTALL_DIR" --with-aio
cd src
$MAKE clean
$MAKE install
Expand All @@ -99,7 +99,7 @@ function build_ior {

function build_pfind {
pushd "$BUILD"/pfind
./prepare.sh
CFLAGS="-fPIC" ./prepare.sh
./compile.sh
ln -sf "$BUILD"/pfind/pfind "$BIN"/pfind
echo "Pfind: OK"
Expand Down
15 changes: 15 additions & 0 deletions src/main.c
View file
Open in desktop
Copy link
Copy Markdown
Contributor

@JulianKunkel JulianKunkel Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, why is the current approach not sufficient? It does check right?

Copy link
Copy Markdown
Author

@jxiong jxiong Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think I can deprecate this patch.

Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,17 @@ static const char * io500_mode_str(io500_mode mode){
}
}

static int should_run_phase(u_phase_t * phase){
if(phase->options){
for(ini_option_t * o = phase->options; o->name != NULL; o++){
if(o->type == INI_BOOL && o->var != NULL && strcmp(o->name, "run") == 0){
return *(int*) o->var;
}
}
}
return 1;
}

int main(int argc, char ** argv){
int mpi_init = 0;
file_out = stdout;
Expand Down Expand Up @@ -378,6 +389,10 @@ int main(int argc, char ** argv){
continue;
}

if(! should_run_phase(phase)){
continue;
}

if(opt.drop_caches && ! (phase->type & IO500_PHASE_DUMMY) ){
DEBUG_INFO("Dropping cache\n");
if(opt.rank == 0)
Expand Down
4 changes: 2 additions & 2 deletions src/phase_dbg.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ static ini_option_t option[] = {


static void validate(void){
if(opt.stonewall != opt.minwrite && opt.rank == 0){
INVALID("stonewall-time < %us\n", opt.minwrite);
if(opt.stonewall < opt.minwrite && opt.rank == 0){
INVALID("stonewall-time: %us < %us\n", opt.stonewall, opt.minwrite);
}
}

Expand Down
3 changes: 2 additions & 1 deletion src/phase_ior.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ typedef struct{
char * transferSize;
char * blockSize;
int verbosity;
int direct;
} opt_ior_easy;

extern opt_ior_easy ior_easy_o;
Expand Down Expand Up @@ -41,7 +42,7 @@ extern opt_ior_rnd ior_rnd4K_o;
extern opt_ior_rnd ior_rnd1MB_o;


void ior_easy_add_params(u_argv_t * argv, int addStdFlags);
void ior_easy_add_params(u_argv_t * argv, int addStdFlags, int direct);
void ior_hard_add_params(u_argv_t * argv);
void ior_rnd4K_add_params(u_argv_t * argv);
void ior_rnd1MB_add_params(u_argv_t * argv);
Expand Down
7 changes: 6 additions & 1 deletion src/phase_ior_easy.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ static ini_option_t option[] = {
{"uniqueDir", "Use unique directory per file per process", 0, INI_BOOL, "FALSE", & ior_easy_o.uniqueDir},
{"run", "Run this phase", 0, INI_BOOL, "TRUE", & ior_easy_o.run},
{"verbosity", "The verbosity level", 0, INI_INT, 0, & ior_easy_o.verbosity},
{"direct", "Use direct IO (posix.odirect)", 0, INI_BOOL, "TRUE", & ior_easy_o.direct},
Copy link
Copy Markdown
Contributor

@JulianKunkel JulianKunkel Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can make an exception for supporting posix.odirect for AIO module.
I suppose you tried using --posix.odirect even though AIO is used?
Let's recap what users expect.

I think what should have worked was sth like:
API = POSIX --aio.max-pending=128 --aio.granularity=8 --posix.odirect
Or did it not?

Copy link
Copy Markdown
Author

@jxiong jxiong Mar 17, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is using POSIX direct IO. What I am trying to do is to enable AIO + direct IO. Something like AIO --aio.max-pending=128 --aio.granularity=8 --posix.odirect would be equivalent but I didn't test it out.

EDITTED: AIO --aio.max-pending=128 --aio.granularity=8 --posix.odirect works with a warning message to complain prefix with --with-aio in the IOR build config.

IO500 version 66fac1a7cca8 (standard)
; WARNING Provided API option --posix.odirect starts with a different prefix and might be not supported - don't worry if you checked!
WARNING Provided API option --posix.odirect starts with a different prefix and might be not supported - don't worry if you checked!
WARNING Provided API option --posix.odirect starts with a different prefix and might be not supported - don't worry if you checked!

{NULL} };

static void validate(void){
Expand Down Expand Up @@ -42,7 +43,7 @@ static void cleanup(void){
}
}

void ior_easy_add_params(u_argv_t * argv, int addStdFlags){
void ior_easy_add_params(u_argv_t * argv, int addStdFlags, int direct){
opt_ior_easy d = ior_easy_o;

u_argv_push(argv, "./ior");
Expand Down Expand Up @@ -82,6 +83,10 @@ void ior_easy_add_params(u_argv_t * argv, int addStdFlags){
if(ior_easy_o.filePerProc){
u_argv_push(argv, "-F"); /* write a separate file per process */
}

if(direct){
u_argv_push(argv, "--posix.odirect"); /* read&write file with direct IO */
}
}

u_phase_t p_ior_easy = {
Expand Down
16 changes: 12 additions & 4 deletions src/phase_ior_easy_read.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
#include <sys/stat.h>
#include <unistd.h>
#include <mpi.h>
#include <string.h>

#include <io500-phase.h>
#include <phase_ior.h>

typedef struct{
int run;
char * api;
int direct;
int iteration;

char * command;
IOR_point_t * res;
Expand All @@ -18,6 +21,8 @@ static opt_ior_easy_read o;
static ini_option_t option[] = {
{"API", "The API to be used", 0, INI_STRING, NULL, & o.api},
{"run", "Run this phase", 0, INI_BOOL, "TRUE", & o.run},
{"direct", "Use direct IO (posix.odirect)", 0, INI_BOOL, NULL, & o.direct},
{"iteration", "Number of iterations", 0, INI_INT, "1", & o.iteration},
{NULL} };


Expand All @@ -30,17 +35,20 @@ static double run(void){
opt_ior_easy d = ior_easy_o;

u_argv_t * argv = u_argv_create();
ior_easy_add_params(argv, 1);
int direct = (o.direct != INI_UNSET_BOOL) ? o.direct : d.direct;
ior_easy_add_params(argv, 1, direct);
u_argv_push(argv, "-r"); /* read existing file */
if (o.iteration != INI_UNSET_INT) {
u_argv_push(argv, "-i");
u_argv_push_printf(argv, "%d", o.iteration);
}
u_argv_push(argv, "-R"); /* verify data read */

u_argv_push_default_if_set_api_options(argv, "-a", d.api, o.api);
u_argv_push(argv, "-O");
u_argv_push_printf(argv, "saveRankPerformanceDetailsCSV=%s/ior-easy-read.csv", opt.resdir);
// u_argv_push_default_if_set(argv, "-U", /* use hints file */
// d.hintsFileName, o.hintsFileName);
// u_argv_push_default_if_set_api_options(argv, "-a", /* backend API */
// d.api, o.api);
// u_argv_push_default_if_set_bool(argv, "--posix.odirect", d.odirect, o.odirect);

o.command = u_flatten_argv(argv);

Expand Down
8 changes: 4 additions & 4 deletions src/phase_ior_easy_write.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
typedef struct{
int run;
char * api;
int direct;

char * command;
IOR_point_t * res;
Expand All @@ -19,6 +20,7 @@ static opt_ior_easy_write o;
static ini_option_t option[] = {
{"API", "The API to be used", 0, INI_STRING, NULL, & o.api},
{"run", "Run this phase", 0, INI_BOOL, "TRUE", & o.run},
{"direct", "Use direct IO (posix.odirect)", 0, INI_BOOL, NULL, & o.direct},
{NULL} };

static void validate(void){
Expand All @@ -29,7 +31,8 @@ static double run(void){
opt_ior_easy d = ior_easy_o;

u_argv_t * argv = u_argv_create();
ior_easy_add_params(argv, 1);
int direct = (o.direct != INI_UNSET_BOOL) ? o.direct : d.direct;
ior_easy_add_params(argv, 1, direct);
u_argv_push(argv, "-w"); /* write file */
u_argv_push(argv, "-D"); /* deadline for stonewall in seconds */
u_argv_push_printf(argv, "%d", opt.stonewall);
Expand All @@ -40,9 +43,6 @@ static double run(void){
u_argv_push_printf(argv, "saveRankPerformanceDetailsCSV=%s/ior-easy-write.csv", opt.resdir);
// u_argv_push_default_if_set(argv, "-U", /* use hints file */
// d.hintsFileName, o.hintsFileName);
// u_argv_push_default_if_set_api_options(argv, "-a", /* backend API */
// d.api, o.api);
// u_argv_push_default_if_set_bool(argv, "--posix.odirect", d.odirect, o.odirect);

o.command = u_flatten_argv(argv);

Expand Down
5 changes: 4 additions & 1 deletion src/phase_ior_rnd_read4k-easywrite.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
typedef struct{
int run;
char * api;
int direct;

char * command;
IOR_point_t * res;
Expand All @@ -18,6 +19,7 @@ static opt_ior_rnd_read o;
static ini_option_t option[] = {
{"API", "The API to be used", 0, INI_STRING, NULL, & o.api},
{"run", "Run this phase", 0, INI_BOOL, "TRUE", & o.run},
{"direct", "Use direct IO (posix.odirect)", 0, INI_BOOL, NULL, & o.direct},
{NULL} };


Expand All @@ -30,7 +32,8 @@ static double run(void){
opt_ior_rnd_read d = o;

u_argv_t * argv = u_argv_create();
ior_easy_add_params(argv, 0);
int direct = (o.direct != INI_UNSET_BOOL) ? o.direct : ior_easy_o.direct;
ior_easy_add_params(argv, 0, direct);
u_argv_push(argv, "-r");
u_argv_push(argv, "-D");
u_argv_push_printf(argv, "%d", opt.stonewall);
Expand Down
4 changes: 3 additions & 1 deletion src/phase_mdworkbench.c
View file
Open in desktop
Copy link
Copy Markdown
Contributor

@JulianKunkel JulianKunkel Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree to this change.

Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,9 @@ void mdworkbench_add_params(u_argv_t * argv, int is_create){
if(! f){
WARNING("Couldn't open mdworkbench-file: %s\n", file);
}else{
fread(& mdtest->rate, sizeof(mdtest->rate), 1, f);
if (fread(& mdtest->rate, sizeof(mdtest->rate), 1, f) != 1) {
WARNING("Failed to read mdworkbench-file: %s\n", file);
}
}
fclose(f);
}
Expand Down