Function/sway label

README.md

@@ -1,4 +1,10 @@
 # Labeler for video labeling
+
+## Key Features
+- **Frame-by-frame video labeling**
+- **Keypoint annotation**
+- **Displacement sway boundary labeling**
+
 ## Web Version
 https://mobility-scooter-project.github.io/labeler/
 - setup by `npm install` in `./web` folder

web/README.md

@@ -68,7 +68,9 @@ After labeling all desired frames of the video, you can hit button "Save labels"
 
 The CSV file would look like this:
 
-![](../docs/images/keypointCSV.png)
+![](../docs/images/keyointCSV.png)
+
+
 
 ##### *Explaination*
 
@@ -98,4 +100,90 @@ Digit of (x<sub>n</sub>,y<sub>n</sub>) where n follows the following notation:
 | 11  | left hip  |
 | 12  | right hip  |
 
+### Step 4: Sway Boundary Labeling
+![](../docs/images/sway-boundaries.png)
+
+#### Mark sway boundaries
+
+1. ##### Set Time Boundaries
+   - Click <kbd>Set Start Time</kbd> at the beginning of the sway motion
+   - Click <kbd>Set End Time</kbd> at the completion of marking all 6 sway positions
+
+2. ##### Mark Sway Positions
+   Place all 6 required points:
+   - **Left Sway** 
+     - Sternum (upper torso)
+     - Umbilicus (lower torso)
+   - **Right Sway**:
+     - Sternum (upper torso)
+     - Umbilicus (lower torso)
+   - **Neutral Position**:
+     - Sternum (center)
+     - Umbilicus (center)
+
+3. ##### Save Boundary Set 
+   Click <kbd>Save Current Sway Boundary</kbd> to store the data
+
+4. ##### Export Data 
+   - Click the dropdown <kbd>Export Annotation Data</kbd>
+   - Select <kbd>Export Sway Boundaries Data</kbd>
+
+### Visual Guide
+![Sway Boundary Marking Demo](docs/videos/sway-boundary.gif)
+
+### Tips
+- Sway points turn green when successfully marked
+- Use the eraser tool to correct misplaced sway points
+- You must mark all 6 points and end time before saving
+- Multiple boundary sets can be saved per video
+![](../docs/videos/sway-boundary.gif)
+
+#### Sway Boundary Features:
+✔ **Visual Feedback System**
+- Real-time display of marked points with coordinate labels
+- Color-coded indicators (Left: Blue, Right: Green, Neutral: Red)
+- Completion checkmarks for finished points
+
+✏ **Smart Editing with Eraser Tool**
+- Precision eraser for individual points
+- Bulk delete for overlapping markers
+
+
+✅ **Validation System**
+- Enforces complete boundary sets:
+  - 2 Left sway positions (sternum/umbilicus)
+  - 2 Right sway positions (sternum/umbilicus)
+  - 2 Neutral positions (sternum/umbilicus)
+- Requires valid time boundaries
+
+#### Export sway boundaries
+Click "Export Annotation Data" to save sway boundaries data to a CSV file with:
+- Start/End timestamps
+- All 6 point boundary with x and y coordinates
+
+![](../docs/images/exportSwayData.png)
+
+
+## Data Export Formats
+
+### Sway Points CSV Format:
+
+| Start_Time | End_Time | Left_Sternum_X | Left_Sternum_Y | Left_Umbilicus_X | Left_Umbilicus_Y | Right_Sternum_X | Right_Sternum_Y | Right_Umbilicus_X | Right_Umbilicus_Y | Neutral_Sternum_X | Neutral_Sternum_Y | Neutral_Umbilicus_X | Neutral_Umbilicus_Y |
+|-----------|---------|----------------|----------------|------------------|------------------|-----------------|-----------------|-------------------|-------------------|-------------------|-------------------|---------------------|---------------------|
+| 655       | 1276    | 454.5          | 197            | 461.5            | 381              | 403.5           | 210             | 471.5             | 391               | 437.5             | 218               | 465.5               | 390                 |
+| 2979       | 4586    | 432.5          | 194            | 427.5            | 392              | 407.5          | 197             | 467.5             | 400               | 464.5             | 209               | 476.5               | 409                 |
+
+
+**Coordinate System:** Origin (0,0) at top-left corner
+
+
+## Troubleshooting
+![](../docs/images/exportSwayDataError.png)
+- If you see the above error message "Export failed: Please save boundaries before exporting", this means:
+    - You haven't saved any sway boundaries yet
+    - First complete and save at least one sway boundary before exporting
 
+- Before saving each sway boundary, ensure:
+    - Both start and end times are set
+    - All 6 required points are marked (Left, Right, and Neutral sway positions for both sternum and umbilicus)
+    - The green checkmarks appear on all point buttons

web/src/App.js

@@ -8,6 +8,7 @@ import Selection from "./components/Selection";
 import { VERSION } from "./version";
 import KeypointList from "./components/KeypointLabel";
 import { keypointsIndex } from "./utils/constant";
+import { SwayPointLabel } from "./components/SwayPointLabel/SwayPointLabel";
 
 const colorLength = 400;
 
@@ -143,6 +144,17 @@ function App() {
     setMarkedKeypoints([]);
     setErrorChooseKeypoint(false);
     setIsRemoveKeypoint(false);
+
+    //sway
+    setSwayPointData([]);
+    setSwayPoints([]);
+    setSelectedSwayPoint(null);
+    setMarkedSwayPoints([]);
+    setIsRemoveSwayPoint(false);
+    setTimeButtonsClicked({
+      start: false,
+      end: false
+    });
   };
 
   const updateLabels = (start, end, label) => {
@@ -365,6 +377,131 @@ function App() {
     setMarkedKeypoints(newMarkedKeypoints);
   };
 
+  // Sway point handling
+  const [swayPoints, setSwayPoints] = useState([]);
+  const [selectedSwayPoint, setSelectedSwayPoint] = useState();
+  const [markedSwayPoints, setMarkedSwayPoints] = useState([]);
+  const [isRemoveSwayPoint, setIsRemoveSwayPoint] = useState(false);
+  const [swayPointData, setSwayPointData] = useState([]);
+  const [startTime, setStartTime] = useState([]);
+  const [endTime, setEndTime] = useState([]);
+  const [timeButtonsClicked, setTimeButtonsClicked] = useState({
+    start: false,
+    end: false
+  });
+
+  // TODO: NEED TO IMPLEMENT
+  const handleCompleteMarkSwayPoint = () => {
+
+    if (swayPoints.length !== 6) {
+      setMessage("Please mark all 6 sway points before saving");
+      return;
+    }
+
+    // Sort points by label
+    const sortedPoints = [...swayPoints].sort((a, b) => a.label - b.label);
+
+    // Store the data
+    setSwayPointData([
+      ...swayPointData,
+      {
+        startTime: startTime,
+        endTime: endTime,
+        points: sortedPoints
+      }
+    ]);
+
+    setSwayPoints([]);
+    setSelectedSwayPoint(undefined);
+    setMarkedSwayPoints([]);
+    setTimeButtonsClicked({
+      start: false,
+      end: false
+    });
+    console.log("The current sway boundary has been saved!!!!");
+  }; 
+
+  const handleSaveSwayBoundaries = () => {
+    if (!swayPointData || swayPointData.length === 0 || !swayPointData[0]?.points) {
+      setMessage("Export failed: Please save boundaries before exporting");
+
+      // Clear message after 3 seconds
+      setTimeout(() => {
+        setMessage("");
+      }, 3000);
+      return;
+    }
+
+    // Create CSV header
+    const csvHeader = [
+      "Start_Time",
+      "End_Time",
+      "Left_Sternum_X", 
+      "Left_Sternum_Y",
+      "Left_Umbilicus_X",
+      "Left_Umbilicus_Y",
+      "Right_Sternum_X",
+      "Right_Sternum_Y",
+      "Right_Umbilicus_X",
+      "Right_Umbilicus_Y",
+      "Neutral_Sternum_X",
+      "Neutral_Sternum_Y",
+      "Neutral_Umbilicus_X",
+      "Neutral_Umbilicus_Y"
+    ];
+
+
+    const dataRows = swayPointData.map((boundary) => {
+      // Sort points by label to ensure consistent order (13-16)
+      const sortedPoints = boundary.points.sort((a, b) => a.label - b.label);
+
+      // Extract coordinates in order: left sternum, right sternum, left umbilicus, right umbilicus
+      const coordinates = sortedPoints.flatMap(point => [point.x, point.y]);
+
+      return [
+        boundary.startTime,  
+        boundary.endTime,   
+        ...coordinates
+      ];
+    });
+
+    // Download CSV with header and all data rows
+    downloadCSV([csvHeader, ...dataRows], `${video}-sway-boundaries.csv`);
+  };
+
+  const handleSetStart = () => {
+    setStartTime(getIndex(time.current, fpsRef.current));
+    setTimeButtonsClicked(prev => ({...prev, start: true}));
+  };
+
+  const handleSetEnd = () => {
+    setEndTime(getIndex(time.current, fpsRef.current));
+    setTimeButtonsClicked(prev => ({...prev, end: true}));
+  };
+
+  console.log("start time: ");
+  console.log(startTime);
+
+  const handleMarkedSwayPoint = (key) => {
+    setSelectedSwayPoint(undefined)
+    setMarkedSwayPoints([...markedSwayPoints, key]);
+  };
+
+  const handleRemoveSwayPoint = (key) => {
+    if (Array.isArray(key)) { // handle multiple overlaping sway points being removed
+      setMarkedSwayPoints(prev => prev.filter(k => !key.includes(k)));
+    } 
+    // If single sway point removed
+    else {
+      setMarkedSwayPoints(prev => prev.filter(k => k !== key));
+    }
+    setTimeButtonsClicked(prev => ({...prev, end: false}));
+  };
+
+  console.log(swayPointData);
+  console.log("...");
+  console.log(swayPoints);
+
   return (
     <div className="container">
       <div className="title-container">
@@ -456,10 +593,18 @@ function App() {
           onErrorMarkedKeypoint={() => setErrorChooseKeypoint(true)}
           isRemoveKeypoint={isRemoveKeypoint}
           onRemoveKeypoint={handleRemoveKeypoint}
+
+          //Swaypoint handler
+          swayPoints={swayPoints}
+          setSwayPoints={setSwayPoints}
+          selectedSwayPoint={selectedSwayPoint}
+          onMarkSwayPoint={handleMarkedSwayPoint}
+          isRemoveSwayPoint={isRemoveSwayPoint}
+          onRemoveSwayPoint={handleRemoveSwayPoint}
         />
         <div className="mid-input-container">
           <label htmlFor="file-upload" className="file-upload-button">
-            Choose Video
+            Upload Video
           </label>
           <input
             id="file-upload"
@@ -470,15 +615,32 @@ function App() {
           />
           <h4>{video}</h4>
           {source !== "" && (
-            <>
-              <button className="save-btn" onClick={handleSaveLabel}>
-                Save labels
+          <div className="dropdown-container">
+            <button className="dropdown-toggle save-btn">
+              Export Annotation Data
+            </button>
+            <div className="dropdown-menu">
+              <button 
+                className="dropdown-item save-btn" 
+                onClick={handleSaveLabel}
+              >
+                Export Labels Data
               </button>
-              <button className="save-btn" onClick={handleSaveKeypoint}>
-                Save Keypoints
+              <button 
+                className="dropdown-item save-btn" 
+                onClick={handleSaveKeypoint}
+              >
+                Export Keypoints Data
               </button>
-            </>
-          )}
+              <button 
+                className="dropdown-item save-btn" 
+                onClick={handleSaveSwayBoundaries}
+              >
+                Export Sway Boundaries Data
+              </button>
+            </div>
+          </div>
+        )}
         </div>
       </div>
 
@@ -516,9 +678,36 @@ function App() {
         </div>
         <div className="hint">Rotation is under construction...</div>
         <div className="hint">Don't refresh midway, no cache yet</div>
+
+        {/* Sway Boundaries Labeling */}
+        <div>
+          <h3 style={{ userSelect: "none"}}>Sway Boundaries</h3>
+          <div className="sway-hint">  
+            <p>  
+              <strong>Tooltip:</strong> Click  
+                <button className="demo-btn">
+                  Set Start Time
+                </button> to begin. All 6 points and timing must be set before saving
+            </p>
+          </div>
+          <SwayPointLabel 
+            onSwayPoint={(k) => {
+              setSelectedSwayPoint(k);
+            }}
+            selected={selectedSwayPoint}
+            marked={markedSwayPoints}
+            onComplete={handleCompleteMarkSwayPoint}
+            isRemoveSwayPoint={isRemoveSwayPoint}
+            setIsRemoveSwayPoint={setIsRemoveSwayPoint}
+            onSetStart={handleSetStart}
+            onSetEnd={handleSetEnd}
+            timeButtonsClicked={timeButtonsClicked}
+          />
+        </div>
       </div>
     </div>
   );
 }
 
 export default App;
+