Feature request - only read specific sections of ini file

[amtssp]

Hi. First thank you for providing shini it is working fine.

I would like to request a new feature, because if we have a long ini file it takes a long time to read, and we often only need to read a specific section of the ini file.
So could you add the possibility to only read the variables in a specified section?

Thanks

[wallyhall]

Absolutely - if this project is seeing this much attention, I'll happily spend the time trying to optimise it!

As soon as possible, I'll investigate general optimisations (to increase the speed overall) and, as you suggest, a specific enhancement to deal with only the section in question.

[amtssp]

Thank you very much.

We have included your "Shini" script in our small free audio player for Raspberry - the piCorePlayer: https://sites.google.com/site/picoreplayer/home

[wallyhall]

That's really cool :-) May I ask for an example copy of the INI file you're parsing, and an example of the operation(s) you're performing on it?

I've made some fairly aggressive changes for bash >= version 3 and zsh (I'm not clear on the version compatibility for zsh), which has pushed the performance to almost 7x faster (in my tests).

Regarding processing a specific section, am I understanding correctly that you'd like to be able to specify shini_parse ... "SOMEBLOCK" so it only parses the content of that specific section?

[wallyhall]

I've included PHP's shipped php.ini as a performance test, and included test.sh which accepts a single argument for the specific INI section to parse. (It simply skips everything else.) Not specifying the argument parses the entire file as normal.

The overall performance increase is very significant with the earlier changes:

Before

time bash test_perf.sh
real    0m16.372s

After

time bash test_perf.sh
real    0m0.789s    

(Note, zsh seems to have the edge on performance in the current implementation.)

Including specifying the section, further gains can be made:

time bash ./test_perf.sh > /dev/null
real    0m0.838s

time bash ./test_perf.sh PHP > /dev/null
real    0m0.396s

time bash ./test_perf.sh opcache > /dev/null
real    0m0.309s

time bash ./test_perf.sh curl > /dev/null
real    0m0.282s

[amtssp]

This is great :-)
Here is our script that among many things also use shini to read a lot of audio DAC.conf files in order to make us able to use the variables in the DACs

#!/bin/sh

# Version: 3.20 2017-03-12
#	Added check for /tmp/dropdown.cfg in pcp_sound_card_dropdown. SBP.
#	Added shini conf parser. SBP.
#	Suppress already mounted message during do_rebootstuff newconfig process. PH.

# Version: 3.11 2017-01-20
#	Fixed the issue with shairport not working when using USB cards. SBP.

# Version: 3.10 2016-12-11
#	Original. SBP.

pcp_debug_log "$0 - Running pcp-soundcard-functions..."

DACLOCATION="/usr/local/etc/pcp/cards"     # <=== Should be in pcp-functions? GE.
#The below line is for development only.
#DACLOCATION="/home/tc/www/cgi-bin/cards"
############

. /usr/local/bin/shini.sh

#========================================================================================
# Check if we use the special Audio kernel or not. As we load special overlays if using Audio kernel
#----------------------------------------------------------------------------------------
pcp_AudioKernel_status() {
	sudo  uname -a | grep pcpAudioCore > /dev/null 2>&1 && KERNELVERSION=Audiokernel || KERNELVERSION=Officialkernel
}
pcp_AudioKernel_status

#========================================================================================
# Disable/re_enable analog audio
#----------------------------------------------------------------------------------------
pcp_disable_analog() {
	sudo sed -i 's/^dtparam=audio=on/#dtparam=audio=on/g' $CONFIGTXT
	sudo sed -i 's/^audio_pwm_mode=2/#audio_pwm_mode=2/g' $CONFIGTXT
}

pcp_re_enable_analog() {
	pcp_disable_analog
	sudo sed -i "/dtparam=audio=on/c\dtparam=audio=on" $CONFIGTXT
	sudo sed -i "/audio_pwm_mode=2/c\audio_pwm_mode=2" $CONFIGTXT
}

#========================================================================================
# Remove Audio dtoverlays
#----------------------------------------------------------------------------------------
pcp_disable_i2s() {
	pcp_re_enable_analog
		# unset DTOVERLAY both the old one that is removed on chooseoutput.cgi and the one from config.cfg to clear all traces of audio dtoverlays
		shini_parse ${DACLOCATION}/${ORIG_AUDIO}.conf
		[ x"" != x"${DTOVERLAY}" ] && sed -i '/dtoverlay='"$DTOVERLAY"'/d' $CONFIGTXT

		shini_parse ${DACLOCATION}/${AUDIO}.conf
		[ x"" != x"${DTOVERLAY}" ] && sed -i '/dtoverlay='"$DTOVERLAY"'/d' $CONFIGTXT
}

#========================================================================================
# Add Audio dtoverlays
#----------------------------------------------------------------------------------------
pcp_enable_i2s() {
	pcp_re_enable_analog
	pcp_selected_soundcontrol
	[ x"$SPARAMS1" != x"" ] && PAR1=','"$SPARAMS1"'' || PAR1="$SPARAMS1"
	[ x"$SPARAMS2" != x"" ] && PAR2=','"$SPARAMS2"'' || PAR2="$SPARAMS2"
	[ x"$SPARAMS3" != x"" ] && PAR3=','"$SPARAMS3"'' || PAR3="$SPARAMS3"
	[ x"$SPARAMS4" != x"" ] && PAR4=','"$SPARAMS4"'' || PAR4="$SPARAMS4"
	[ x"$SPARAMS5" != x"" ] && PAR5=','"$SPARAMS5"'' || PAR5="$SPARAMS5"

	[ x"$DTOVERLAY" != x"" ] && sudo echo dtoverlay=$DTOVERLAY$PAR1$PAR2$PAR3$PAR4$PAR5  >> $CONFIGTXT
}

#========================================================================================
# Enable/disable HDMI settings in config.txt
#----------------------------------------------------------------------------------------
pcp_disable_HDMI() {
	sed -i '/hdmi_drive=2/d' $CONFIGTXT
	sed -i '/hdmi_force_hotplug=1/d' $CONFIGTXT
	sed -i '/hdmi_force_edid_audio=1/d' $CONFIGTXT
}

pcp_enable_HDMI() {
	pcp_disable_i2s
	sudo echo hdmi_drive=2 >> $CONFIGTXT
	sudo echo hdmi_force_hotplug=1 >> $CONFIGTXT
	sudo echo hdmi_force_edid_audio=1 >> $CONFIGTXT
	sudo amixer cset numid=3 2 >/dev/null 2>&1
}

#========================================================================================
# Section that controls loading of DAC overlays
#----------------------------------------------------------------------------------------
pcp_read_chosen_audio() {
	# if noumount, disk is probably already mounted....supress output
	[ "$1" == "noumount" ] && pcp_mount_mmcblk0p1 > /dev/null 2>&1 || pcp_mount_mmcblk0p1
	pcp_disable_HDMI
	pcp_disable_i2s
	pcp_enable_i2s
	# This is to allow no unmount of mmcblk0p1 during booting with newconfig.cfg
	[ "$1" != "noumount" ] && pcp_umount_mmcblk0p1
}

#========================================================================================
# Control of sound card routines:
#  - Needs to be populated for each type of soundcard.
#  - Some are without filter options.
#  - USB cards will probably never be supported this way as they are so different.
#----------------------------------------------------------------------------------------
# HELP - To find the controls:
#  - amixer -c 0 scontrols
#  - aplay -l
#----------------------------------------------------------------------------------------
pcp_generic_card_control() {
	case "$GENERIC_CARD" in
		TI51XX)
			amixer -c $CARD scontrols | grep -q 'DSP Program' && DSPfound="yes" || DSPfound="no" 
			if [ "$DSPfound" = "yes" ]; then
				DSP="DSP Program,0"
				FILTER1="Low latency IIR with de-emphasis"
				FILTER2="FIR interpolation with de-emphasis"
				FILTER3="High attenuation with de-emphasis"
				FILTER4="Fixed process flow"
				FILTER5="Ringing-less low latency FIR"
			fi
			ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Right: Playback" | awk '{ print $5 }' | tr -d "[]%")
			ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Right: Playback" | awk '{ print $6 }' | tr -d "[]")
			ACTUAL_FILTER=$(amixer -c $CARD sget 'DSP Program,0' | grep "Item0:" | awk '{ print $2 }' | tr -d "'")

			case "$DSPFILTER" in
				FILTER1) FILTER="$FILTER1" ;;
				FILTER2) FILTER="$FILTER2" ;;
				FILTER3) FILTER="$FILTER3" ;;
				FILTER4) FILTER="$FILTER4" ;;
				FILTER5) FILTER="$FILTER5" ;;
			esac

			# Logic to make checked radiobuttons - needs to clear it otherwise we have two FILTERS_CHECK checked.
			FILTER1_CHECK=""
			FILTER2_CHECK=""
			FILTER3_CHECK=""
			FILTER4_CHECK=""
			FILTER5_CHECK=""

			case "$ACTUAL_FILTER" in
				Low) FILTER1_CHECK="checked" ;;
				FIR) FILTER2_CHECK="checked" ;;
				High) FILTER3_CHECK="checked" ;;
				Fixed) FILTER4_CHECK="checked" ;;
				Ringing-less) FILTER5_CHECK="checked" ;;
			esac
		;;

		HIFIBERRY_AMP)
			ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono:" | awk '{ print $3 }' | tr -d "[]%")
		;;

		ONBOARD)
			ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $4 }' | tr -d "[]%")
			ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $5 }' | tr -d "[]")
		;;

#		ES9023)
#			CARD="ALSA"
#			SSET="PCM"
#			ACTUAL_VOL=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $4 }' | tr -d "[]%")
#			ACTUAL_DB=$(amixer -c $CARD sget $SSET | grep "Mono: Playback" | awk '{ print $5 }' | tr -d "[]")
#			TEXT="Choosing 384k will enable 352k and 384k sample rates. Selecting bclk_ratio_int_div will use bclk_ratio=50 for 16/24bps and bclk_ratio=100 for 32bps media when sample rate is a multiple of 8kHz and less than 192kHz. Which causes the selection of the 19M2 OSC as the parent for the PCM clock with an integer divider, rather than PLLD with fractional divider and MASH noise shaping."
#			PARAMS1="384k"
#			PARAMS2="bclk_ratio_int_div"
#		;;
	esac
}

#========================================================================================
# This will get us all available AUDIO DACs
#----------------------------------------------------------------------------------------
AUDIO_OPTIONS_EXTRA=$(ls "$DACLOCATION" | grep ".conf" | awk -F'.conf' '{ print $1 }')

#========================================================================================
# Section that populate sound card drop-down list on Squeezelite.cgi
#----------------------------------------------------------------------------------------
pcp_sound_card_dropdown() {
	pcp_blank_audio_conf
	. $CONFIGCFG
	if [ -e /tmp/dropdown.cfg ]; then
		sudo sed -i 's/:selected:/:notselected:/' /tmp/dropdown.cfg
		sudo sed -i "/${AUDIO}:/s/:notselected:/:selected:/" /tmp/dropdown.cfg

	else
		CONF_AUDIO="$AUDIO"
		pcp_blank_audio_conf
		for i in $AUDIO_OPTIONS_EXTRA; do
			AUDIO=$i
			pcp_soundcontrol
			CHECKED="notselected"
			[ "$AUDIO" = "$CONF_AUDIO" ] && CHECKED="selected"
			if [ x"$LISTNAME" != x ]; then
			sudo echo "${AUDIO}:${CHECKED}:${LISTNAME}:${RPI_MODEL}" >> /tmp/dropdown.cfg
			fi
		done
	fi
}

#========================================================================================
# This will get us available options from the DAC
#----------------------------------------------------------------------------------------
old_pcp_soundcontrol() {
	pcp_blank_audio_conf
	echo "$AUDIO_OPTIONS_EXTRA" | grep -q "$AUDIO" && . "$DACLOCATION"/"$AUDIO".conf
}

# Declare a handler for parsed variables.  This is required.
__shini_parsed() {
#	printf "  %s_%s='%s'\n" "$1" "$2" "$3"
	eval \ $1_$2='"$3"'

	if [ "$KERNELVERSION" = "Audiokernel" ]; then
		if [ "$1" = "PCPAUDIOCORE" ]; then
#			printf "  %s_%s='%s'\n" "$1" "$2" "$3"
			eval \ $2='"$3"'
		fi
	fi

	if [ "$KERNELVERSION" = "Officialkernel" ]; then
		if [ "$1" = "PCPCORE" ]; then
#			printf "  %s_%s='%s'\n" "$1" "$2" "$3"
			eval \ $2='"$3"'
		fi
	fi

	if [ "$1" = "COMMON" ]; then
#		printf "  %s_%s='%s'\n" "$1" "$2" "$3"
		eval \ $2='"$3"'
	fi
}

pcp_soundcontrol() {
	# Parse
	shini_parse ${DACLOCATION}/${AUDIO}.conf
}

#========================================================================================
# This will get us available options from the selected DAC
#----------------------------------------------------------------------------------------
pcp_selected_soundcontrol() {
	pcp_blank_audio_conf
	. "$CONFIGCFG"
	shini_parse ${DACLOCATION}/${AUDIO}.conf
}

pcp_blank_audio_conf() {
	# Remove left-over from previous audio DAC
	CARD=""
	OUTPUT=""
	ALSA_PARAMS=""
	GENERIC_CARD=""
	CONTROL_PAGE=""
	LISTNAME=""
	DTOVERLAY=""
	SSET=""
	PARAMS1=""
	PARAMS2=""
	PARAMS3=""
	PARAMS4=""
	PARAMS5=""
	RPI_MODEL=""
	I=1
	while true; do
		eval TEST="\${TEXT${I}}"
		if [ x"$TEST" != x ]; then 
			eval TEXT${I}=""
			I=$((I+1))
		else
			break
		fi
	done
	}

Here is a sample DAC.conf file

[COMMON]
CARD="sndrpihifiberry"
LISTNAME="HiFiBerry DAC+ Light"
OUTPUT="hw:CARD=sndrpihifiberry"
ALSA_PARAMS="80:4::1"
RPI_MODEL="HAT_ALL"

[PCPAUDIOCORE]
GENERIC_CARD="ES9023"
DTOVERLAY="hifiberry-dacpluslight-es9023-audio"
TEXT1="<b>384k</b> will enable 352k and 384k sample rates."
TEXT2="<b>bclk_ratio_int_div</b> will use bclk_ratio=50 for 16/24bps and bclk_ratio=100 for 32bps media when sample rate is a multiple of 8kHz and less than 192kHz. Which causes the selection of the 19M2 OSC as the parent for the PCM clock with an integer divider, rather than PLLD with fractional divider and MASH noise shaping."
PARAMS1=""
PARAMS2="384k"
PARAMS3="bclk_ratio_int_div"
CONTROL_PAGE="soundcard_control.cgi"

[PCPCORE]
DTOVERLAY="hifiberry-dac"
CONTROL_PAGE=""

Please note that we are using ash #!/bin/sh as we are building on piCore which is using busybox

[amtssp]

Yes we would like to be able to only parse a specific part of the ini file (for other parts of our player) - so your additions are highly appreciated.

[amtssp]

Sorry - but the new shini script do not run on our system.
This function stalls it all:

shini_regex_replace()
{
    if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then
        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
        return 0
    fi

    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

If I change it to this it runs, but ..:

shini_regex_replace()
{
#    if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then
#        [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1"
#        return 0
#    fi

    shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")"  # If you have isses on older systems,
    # it may be the non-newer POSIX compliant sed.
    # -E should be enabling extended regex mode portably.
}

[wallyhall]

Ah, I've not tested t under ash. Which version specifically are you using? I'll see if I can fix the incompatibility.

[wallyhall]

Ok. Could you let me know you shell version? (Specifically the shell version the script is being ran under.)

…

Sent from my iPhone

On 6 May 2017, at 13:43, amtssp ***@***.***> wrote: Sorry - but the new shini script do wont run on our system. This function stalls it all: shini_regex_replace() { if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1" return 0 fi shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")" # If you have isses on older systems, # it may be the non-newer POSIX compliant sed. # -E should be enabling extended regex mode portably. } If I change it to this it runs, but ..: shini_regex_replace() { # if [ "${BASH_VERSINFO}" -ge 3 ] || [ -n "$ZSH_VERSION" ]; then # [[ "$1" =~ $2 ]] && shini_retval=${BASH_REMATCH[1]} || shini_retval="$1" # return 0 # fi shini_retval="$(printf '%s' "$1" | sed -E "s/$2/\1/")" # If you have isses on older systems, # it may be the non-newer POSIX compliant sed. # -E should be enabling extended regex mode portably. } — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[amtssp]

Sorry for the delay.
Is this what you were asking for?

tc@piCorePlayer:~$ ash --help
BusyBox v1.24.2 (2016-09-08 12:09:43 UTC) multi-call binary.

[amtssp]

Hi. Thanks for the changes now shini parse the file fine using busybox ash. It is still very slow (on a Raspberry pi Zerro Wireless) here is the result:

tc@piCorePlayer:~/www/cgi-bin$ time ash ./test_perf.sh > /dev/null
Command exited with non-zero status 1
real    2m 1.38s
user    0m 25.24s
sys     0m 52.27s

Do you think that you can improve the speed on ash as well? I noticed this discussion here:
[http://stackoverflow.com/questions/21010882/how-to-match-regexp-with-ash]

I don't know if the last answer on that page will improve speed?

[wallyhall]

No problem, I realised there was a bug breaking the "fallback" functionality (which as you rightly notice, is extremely slow even on the best of hardware), which I've patched.

I too had read that article, but haven't had time to confirm it works yet. I'll give it a whirl this weekend and let you know. (I'm able to test locally against dash, which I believe is a fork of ash - and certainly seems to exhibit the same behaviour as you're seeing.)

I'll let you know ASAP. Thanks for your patience... :-)

[amtssp]

Hi there. Is there anything I can do to help you in the process of making shini faster using busybox or ash?

[wallyhall]

I've had an extensive think about the problem and tried to debug where the duration is being spent.

I suspected - and I believe have now proved beyond doubt - that the slowness comes from calling grep and sed (which I'm using as a fallback for non bash/zsh, which both have regex match and replacement built in. ksh has regex matching, but is relying on sed for replacements.)

Thus the performance is significantly faster in zsh and bash, slower in ksh and abysmal in anything else.

I'm not that familiar with ash, so if you can think of any way to do what I'm achieving with those grep and sed calls in a more "builtin" manner, that'd help more than a lot.

I've spent some time trying to use named pipes, so there's only one grep/sed process spawned at the start for each of the regexes used - but as you can imagine it's super hacky (you need around 22 named pipes and 11 children processes to get anywhere close).

Otherwise I'm afraid I'm out of ideas, except to suggest switching to another library which you could call via anything else you have available (Python/PHP) or knock something up in C. (I'm happy to give that a go if you're willing to compile+ship it with your project. Bit of an ache though!)

There might be some further micro-optimisations around the order of execution etc, but I don't think it'll make enough of a difference for as long as grep/sed are being used.

FYI - these are the performance gains by using builtins from other shells. Note it is still sped up by searching a specific section rather than the entire file - but still awfully slow in dash:

$ wc -l tests/php.ini 
1917 tests/php.ini

$ time zsh ./test_perf.sh > /dev/null
real    0m0.595s

$ time bash ./test_perf.sh > /dev/null
real    0m0.838s

$ time ksh ./test_perf.sh > /dev/null
real    0m2.901s

$ time zsh ./test_perf.sh opcache > /dev/null
real    0m0.237s

$ time bash ./test_perf.sh opcache > /dev/null
real    0m0.313s

$ time ksh ./test_perf.sh opcache > /dev/null
real    0m0.543s

$ time dash test_perf.sh > /dev/null
real	0m17.930s

$ time dash test_perf.sh opcache > /dev/null
real	0m5.839s

[amtssp]

Thanks for trying.
I noticed this on the busybox web page: https://www.busybox.net/BusyBox.html

expr

    expr EXPRESSION

    Print the value of EXPRESSION to stdout

    EXPRESSION may be:

            ARG1 | ARG2     ARG1 if it is neither null nor 0, otherwise ARG2
            ARG1 & ARG2     ARG1 if neither argument is null or 0, otherwise 0
            ARG1 < ARG2     1 if ARG1 is less than ARG2, else 0. Similarly:
            ARG1 <= ARG2
            ARG1 = ARG2
            ARG1 != ARG2
            ARG1 >= ARG2
            ARG1 > ARG2
            ARG1 + ARG2     Sum of ARG1 and ARG2. Similarly:
            ARG1 - ARG2
            ARG1 * ARG2
            ARG1 / ARG2
            ARG1 % ARG2
            STRING : REGEXP         Anchored pattern match of REGEXP in STRING
            match STRING REGEXP     Same as STRING : REGEXP
            substr STRING POS LENGTH Substring of STRING, POS counted from 1
            index STRING CHARS      Index in STRING where any CHARS is found, or 0
            length STRING           Length of STRING
            quote TOKEN             Interpret TOKEN as a string, even if
                                    it is a keyword like 'match' or an
                                    operator like '/'
            (EXPRESSION)            Value of EXPRESSION

    Beware that many operators need to be escaped or quoted for shells. Comparisons are arithmetic if both ARGs are numbers, else lexicographical. Pattern matches return the string matched between \( and \) or null; if \( and \) are not used, they return the number of characters matched or 0.

So can the expr command be useful?