7.2 Write a program that prompts for a file name, then opens that file and reads through the file, looking for lines of the form: X-DSPAM-Confidence: 0.8475 Count these lines and extract the floating point values from each of the lines and compute the average of those values and produce an output as shown below. Do not use the sum() function or a variable named sum in your solution. You can download the sample data at http://www.py4e.com/code3/mbox-short.txt when you are testing below enter mbox-short.txt as the file name.

Python File Processing Example: Calculating Average Spam Confidence

When working with email systems or large datasets, programmers often need to analyze log files to extract useful information. Python provides powerful tools for reading and processing files efficiently. One common learning exercise is analyzing an email log file and calculating statistical values from it.

In this tutorial, we will learn how to read a text file, search for specific lines, extract numeric values, and calculate the average of those numbers. The program processes a file called mbox-short.txt, which contains email data including spam confidence values.

This type of exercise is commonly used in learning resources such as Python for Everybody, which helps beginners understand file handling, loops, and string processing in Python.

---

Problem Description

The objective of this program is simple:

1. Ask the user for the file name.

2. If the user presses Enter without typing a file name, the program automatically uses mbox-short.txt.

3. Open the file and read it line by line.

4. Find lines that start with the text "X-DSPAM-Confidence:".

5. Extract the numeric value from those lines.

6. Convert the extracted text into a floating-point number.

7. Calculate the average of all those numbers.

8. Print the final average spam confidence value.

This example helps students understand how to analyze structured text data using Python.

---

Python Program

Below is the complete Python program:

fname = input("Enter file name: ")
if len(fname) == 0:
    fname = 'mbox-short.txt'

fh = open(fname)

count = 0
tot = 0
ans = 0

for line in fh:
    if not line.startswith("X-DSPAM-Confidence:") :
        continue
    
    count = count + 1
    num = float(line[21:])
    tot = num + tot

ans = tot / count

print ("Average spam confidence:", ans)

---

Understanding the Program Step by Step

Let us examine how this program works in detail.

---

1. Asking the User for File Name

The program begins by asking the user to enter the name of the file.

fname = input("Enter file name: ")

Example input:

Enter file name: mbox-short.txt

The file name entered by the user is stored in the variable fname.

---

2. Handling Empty Input

Sometimes the user may simply press Enter without typing anything. To handle this situation, the program checks whether the length of the input is zero.

if len(fname) == 0:
    fname = 'mbox-short.txt'

If the user provides no input, the program automatically sets the default file name as mbox-short.txt.

This ensures the program still runs even if the user does not enter a file name.

---

3. Opening the File

Next, the program opens the file.

fh = open(fname)

The open() function allows Python to access the contents of the file.

The variable fh now represents the file object and can be used to read the file line by line.

---

4. Creating Variables

Before processing the file, the program initializes three variables:

count = 0
tot = 0
ans = 0

These variables serve different purposes.

• count keeps track of how many spam confidence values are found.

• tot stores the total sum of all spam confidence values.

• ans will store the final average.

Initializing variables is important because it prepares the program for calculations.

---

5. Reading the File Line by Line

The program uses a loop to read each line from the file.

for line in fh:

Each iteration processes one line of the file.

Example lines in the file might look like this:

X-DSPAM-Confidence: 0.8475
X-DSPAM-Confidence: 0.6178
X-DSPAM-Confidence: 0.6961

These lines contain spam confidence values that we want to analyze.

---

6. Filtering Required Lines

Not all lines in the file contain spam confidence values. Therefore, the program checks whether the line starts with the required text.

if not line.startswith("X-DSPAM-Confidence:"):
    continue

This statement performs two actions.

First, it checks whether the line begins with the string "X-DSPAM-Confidence:".

Second, if the line does not match this condition, the continue statement skips the line and moves to the next iteration of the loop.

This ensures that only relevant lines are processed.

---

7. Counting Matching Lines

If the line contains the required text, the program increments the counter.

count = count + 1

This helps keep track of how many spam confidence values were found in the file.

---

8. Extracting the Numeric Value

Next, the program extracts the numeric value from the line.

num = float(line[21:])

Here, line[21:] means starting from the 21st character of the string until the end.

Example line:

X-DSPAM-Confidence: 0.8475

The first 21 characters contain the label text. Everything after that is the numeric value.

Result:

0.8475

The float() function converts the string value into a floating-point number so it can be used in calculations.

---

9. Adding to the Total

The extracted value is then added to the total.

tot = num + tot

This step keeps accumulating all spam confidence values found in the file.

For example:

0.8475 + 0.6178 + 0.6961

---

10. Calculating the Average

After the loop finishes processing the entire file, the program calculates the average.

ans = tot / count

The formula used is:

Average = Total Sum ÷ Number of Values

This gives the average spam confidence value across all matching lines.

---

11. Printing the Result

Finally, the program prints the result.

print("Average spam confidence:", ans)

Example output:

Average spam confidence: 0.7507185185185187

This value represents the average spam confidence found in the email dataset.

---

Why This Program is Important

This example teaches several fundamental Python concepts.

File Handling

Reading files is essential for processing logs, datasets, and reports.

String Processing

The program demonstrates how to extract useful data from text.

Conditional Statements

Filtering specific lines allows efficient data analysis.

Numerical Calculations

Converting text into numeric values enables statistical computations.

---

Real-World Applications

Programs like this are used in many real-world situations, including:

• Email spam detection systems

• Log file analysis

• Data science preprocessing

• Cybersecurity monitoring

Learning these techniques helps programmers build practical data-processing tools.

---

Possible Improvements

Although this program works well, it can be improved.

For example, instead of slicing strings manually, Python's split() function could be used to extract the number more clearly.

Example:

value = line.split(":")[1]
num = float(value)

This makes the program easier to read and maintain.

---

Conclusion

In this tutorial, we explored how Python can be used to analyze a text file and compute the average spam confidence value. The program demonstrates several key programming concepts including file handling, loops, conditional filtering, string slicing, and numerical calculations.

By mastering these techniques, beginners can build a strong foundation in Python programming. Exercises like this also prepare learners for more advanced tasks such as data analysis, automation, and machine learning.

Practicing file-processing programs regularly will greatly improve your ability to solve real-world programming problems efficiently.