E-Mail Problems Plague Campus

Susan M. Groppi '98 was in for a surprise when she logged into her e-mail account last month.

She had six new messages, but they had each been sent more than two weeks earlier.

"I received approximately six e-mails that had been delayed several weeks in delivery, including one announcement of a surprise birthday party for a close friend," Groppi says.

About 50 students had similar problems, receiving e-mail messages through the Faculty of Arts and Sciences' (FAS) mail server several weeks after they were initially sent.

The College's e-mail system has been less than stellar this semester, with students and faculty complaining about numerous system failures and a waiting time that sometimes exceeded 25 minutes to log in.

Professor of Classics and History Christopher P. Jones took the FAS to task at last month's Faculty meeting.

"Many of us who use e-mail regularly...know that the situation is not well. I would like to know...what amelioration we can hope for," Jones said.

Three hours later when Jones tried to e-mail his comments to The Crimson, the system wouldn't work.

He telephoned instead.

"It does seem to me that the Harvard people should be able to get their act together," he said in the interview.

The people responsible for providing the College with e-mail, Internet and other network services are the Harvard Arts and Sciences Computer Services (HASCS)--the support arm of the FAS.

HASCS employees have worked hard to fix the problems. So hard, in fact, that they often have little time to do anything else.

A two-month Crimson investigation, including interviews with about 100 faculty and staff, found that HASCS's focus on simply keeping the system operating has prevented it from creating and implementing a larger vision for information technology in the College.

Unprecedented Usage

At the heart of HASCS's problems is the increasing demand being placed on the system every day.

"Faculty and student demand seems to be unsatiable," says Franklin M. Steen, the director of HASCS.

For instance, the mail server processed 312,000 e-mail messages on Nov. 21, and that day saw more than 1,000 users logged on to the system concurrently, Steen says.

By comparison, the system was processing an average of 40,000 to 50,000 e-mail messages a day when Steen became director in the spring of 1994.

"Two years ago, the average user went to the Science Center every day to check their mail," says Rick Osterberg '96, HASCS's coordinator of residential computing support. "The system got remarkably little use. The average user now checks mail every two hours."

Two years ago, the prevalence of e-mail was measured in number of student accounts, a criterion which is now void because HASCS precreates accounts for all incoming first years.

Now, e-mail's prevalence is measured in the number of student dorm network connections.

The Class of 1997 was the first to receive room connections during their first year.

"We're swimming upstream with a torrent coming down," says Dean of the Division of Applied Sciences Paul C. Martin '53, the chair of the FAS Committee on Information Technology (IT).

To meet the increasing demand, Harvard spent $300,000 this summer to purchase new Digital servers--the UNIX machines users log into to use Pine--including a new FAS machine and a new mail server.

While expanding the system to deal with the massive growth in usage, HASCS introduced new problems by installing the new servers shortly after orientation week.

The machines worked well independently, but they were not communicating with each other effectively, a necessary requirement for the smooth functioning of the e-mail system.

Throughout the semester, the channel between the mail server and the FAS machine clogged considerably during peak use, leading to slow mail delivery and long log-in waits.

"The new machines are not near capacity," Osterberg said last month. "They're having difficulty talking to the mail server."

The problems are caused indirectly by the system architecture chosen by Harvard.

Under this system, every time someone checks mail with Pine, the program used most by faculty and students, it transfers their mail from the mail server to the log-in machine.

In contrast, Eudora, another program used to read e-mail, does not require users to log into a UNIX server but accesses the mail server directly and downloads mail onto the user's desktop computer.

But only about 15 percent of users at Harvard use the Eudora program, according to a survey by HASCS last spring, because it does not allow users to save messages in a central area, where they can later be accessed from another remote location.

Dean of the College Harry R. Lewis '68, a computer science professor, says the College does not want to restrict students' software options.

"The major recommendations of the FAS IT reports...have not been accompanied by descriptions of specific hardware and software platforms," Lewis says. "One size does not fit all in this environment."

Working out the Kinks

HASCS's UNIX staff has been working long hours this term trying to work out the kinks in the new system, but until recently things have only gotten worse.

"When you have a little bit of noise in a system, things start to go wrong everywhere," says McKay Assistant Professor of Computer Science Margo I. Seltzer '83, an expert in system architecture.

In October, HASCS added an accelerator card to the FAS machine in order to make it faster and upgraded the communication channel between the FAS machine and the mail server, according to Osterberg.

But problems persisted, and Jones pressed Dean of the Faculty Jeremy R. Knowles for an improvement on Tuesday, Nov. 12. Ironically, the system hit rock bottom at the end of that week.

Two days after the Faculty meeting, the UNIX group noticed that the mail server was using only half its memory, Osterberg says.

Technicians brought the system down to install new memory. When that failed they reconfigured the system software to fix the memory problem, according to Osterberg. But then the machine started to process the backlogged mail and the system again became absurdly slow.

A day later, HASCS's technicians realized that part of the problem was being caused by the Eudora program. Although fewer students use Eudora, the way they used it was crippling the system.

Many Eudora users had the program set to automatically check their mail every minute or less, placing an overwhelming demand on the system.

"If you're a Eudora user and you're checking your mail every hour, we love you," Osterberg says. "If you're checking you're mail every minute, you're worse than a Pine user."

HASCS then revised the setup of the machines to only allow Eudora users to check their mail every 15 minutes.

As a result, some of the kinks have been ironed out.

"E-mail is, on the whole, stable," Osterberg said on Monday, although he added that there were still occasional problems with the load.

Obstacles

HASCS, which is charged with maintaining the computer network, has a difficult mission.

The organization has come a long way from three years ago, when then acting director Richard S. Steen (no relation to Frank) described HASCS in a confidential memo obtained then by The Crimson as "a ship taking on water faster than it can bail it out."

In the memo to the FAS IT Committee, then chaired by Martin and Lewis, Richard Steen cited three problems: budgetary constraints, staff and space shortages and poor working conditions.

The problems plaguing HASCS then have mostly been solved.

HASCS' budget has risen to $4.8 million for 1997, compared to $2.3 million in 1994, according to Martin.

And its staff has increased from 18 full time employees in 1994 to 32 today, according to Frank Steen.

"We have an adequate level of staffing and excellent people working for us," Steen says.

But HASCS now struggles with staff turnover--six people have left its UNIX group in the past two years, although one later returned. The UNIX group is responsible for maintaining and installing the servers that have been so problematic this year.

"Salary was not the main issue for any of the people who left," Steen says. "Most people left to pursue interesting jobs elsewhere."

However, jobs in HASCS pay significantly less than those in industry. And at least two of the six people to leave took jobs in industry.

The UNIX group now has six fulltime employees, who are on-call 24 hours a day. When the network crashed on Nov. 14, members of the UNIX staff worked most of the night to fix the problems, according to Osterberg.

The Crimson was unable to speak with any current UNIX staff members because Steen does not allow employees other than himself or Osterberg to speak with the press.

Solving the Problems

HASCS employees have been working feverishly to fix the problems with the system.

"Our staff has been working day and night to get the new equipment working," Steen says. "It took longer than we expected [but] the load was also greater than expected."

The IT committee also commended the job done by HASCS staff.

"The HASCS staff is a resilient group," reads the committee's September report. "They enjoy working hard and often work late at night and early in the morning when there would be little to see through their windows--if they had them."

Already this year, HASCS has made substantial improvements. It broke its own record for getting student accounts operating at the beginning of the year, by connecting 4,500 students in three weeks.

Computer support has also improved, according to Osterberg, who is responsible for coordinating the user assistants who help undergraduates and graduate students set up network connections and handle other e-mail and networking problems.

HASCS employs 52 students as UAs this year, compared to only 10 about five years ago, Osterberg says.

"Our residential support is on the cutting edge," Osterberg says. "Harvard's model is the second best in the nation only to Stanford which had a 10-year [lead]."

But HASCS will have to continue to work hard on the network because this fall's changes do not represent a long-term solution.

As a result of the increasing usage, Osterberg describes the newly, installed servers as "a medium term solution" which should last until the spring.

HASCS is considering major changes in the architecture of the e-mail system in order to achieve the expectation that "you should always be able to get your mail and always get it delivered immediately, you shouldn't have to wait to log-on and the system won't go down," Osterberg says. "That's the expectation we work to maintain."

The organization is considering instituting IMAP protocol which connects directly to the mail server but downloads only what a user needs, according to Osterberg.

IMAP, unlike Eudora, allows users to save their messages on a central server and retrieve them from any remote location.

Is There a Vision?

Despite the growth of its staff, HASCS's focus on simply keeping the system operating has prevented it from creating and following a larger vision for information technology in the College.

"Fixing the mail server has been very demanding work," Groppi says. "HASCS needs to be better supported by the University. There's a lot more stuff they could be doing if they had more resources."

The turnover in the UNIX group has left several projects undone and made the installation of the new FAS and mail servers particularly difficult.

Faculty and students have criticized HASCS for installing the new machines at the start of school, when the load increases the most. Installing the machines at the start of the summer would have been far more efficient, faculty and staff say.

But Steen says the machines were not installed earlier because the UNIX group had to stabilize before it could carry out the planning and installation.

The turnover has led to a loss of institutional memory and forced employees to acclimate themselves to Harvard's system.

"Turnover makes anything difficult," Osterberg says. "You lose your internal consistency when you bring new people in and they have to catch up and we have to get them used to our needs and our environment."

This turnover, and the unprecedented usage, has made coordinating ongoing projects and providing consistent service difficult.

"HASCS has to do a lot of firefighting," says Seltzer, a member of the IT committee.

For instance, HASCS has planned to work on authentication systems so that the College can provide grades and other sensitive material via the World Wide Web, but "we've had problems with UNIX staff turnover," Steen says.

HASCS has also worked on improving security on the FAS network but has thus far been unable to complete the job, according to Steen.

Presently when someone enters their password to log into their machine, it passes clearly over the network.

"With the right equipment, I could sit in the middle and watch your whole telnet connection with FAS, including what mail you send," Osterberg says.

HASCS's security efforts have emphasized to students the illegality of breaking into other people's accounts.

People who are caught are sent to the Administrative Board for disciplinary hearings, according to Steen. But he declined to say how many students are Ad Boarded each year because of computer violations.

The focus on keeping the system running also prevents HASCS from providing services including group accounts, mailing lists and Web pages to student groups. This service is instead handled by the Harvard Computer Society (HCS), which now serves 170 student organizations for free.

Groppi, who is HCS's director of account services, describes HASCS's refusal to help student groups as "unforgivable."

Groppi and HCS President Daniel A. Lopez '97 say HCS is happy to provide services to student groups and plans to continue its efforts.

But they say HASCS should be responsible for providing the services.

"We don't mind doing it, but Harvard should take a stand and make this an official service," Lopez says. "The need is big enough."Crimson File PhotoFRANKLIN M. STEEN, Director of HASCS

"We're swimming upstream with a torrent coming down," says Dean of the Division of Applied Sciences Paul C. Martin '53, the chair of the FAS Committee on Information Technology (IT).

To meet the increasing demand, Harvard spent $300,000 this summer to purchase new Digital servers--the UNIX machines users log into to use Pine--including a new FAS machine and a new mail server.

While expanding the system to deal with the massive growth in usage, HASCS introduced new problems by installing the new servers shortly after orientation week.

The machines worked well independently, but they were not communicating with each other effectively, a necessary requirement for the smooth functioning of the e-mail system.

Throughout the semester, the channel between the mail server and the FAS machine clogged considerably during peak use, leading to slow mail delivery and long log-in waits.

"The new machines are not near capacity," Osterberg said last month. "They're having difficulty talking to the mail server."

The problems are caused indirectly by the system architecture chosen by Harvard.

Under this system, every time someone checks mail with Pine, the program used most by faculty and students, it transfers their mail from the mail server to the log-in machine.

In contrast, Eudora, another program used to read e-mail, does not require users to log into a UNIX server but accesses the mail server directly and downloads mail onto the user's desktop computer.

But only about 15 percent of users at Harvard use the Eudora program, according to a survey by HASCS last spring, because it does not allow users to save messages in a central area, where they can later be accessed from another remote location.

Dean of the College Harry R. Lewis '68, a computer science professor, says the College does not want to restrict students' software options.

"The major recommendations of the FAS IT reports...have not been accompanied by descriptions of specific hardware and software platforms," Lewis says. "One size does not fit all in this environment."

Working out the Kinks

HASCS's UNIX staff has been working long hours this term trying to work out the kinks in the new system, but until recently things have only gotten worse.

"When you have a little bit of noise in a system, things start to go wrong everywhere," says McKay Assistant Professor of Computer Science Margo I. Seltzer '83, an expert in system architecture.

In October, HASCS added an accelerator card to the FAS machine in order to make it faster and upgraded the communication channel between the FAS machine and the mail server, according to Osterberg.

But problems persisted, and Jones pressed Dean of the Faculty Jeremy R. Knowles for an improvement on Tuesday, Nov. 12. Ironically, the system hit rock bottom at the end of that week.

Two days after the Faculty meeting, the UNIX group noticed that the mail server was using only half its memory, Osterberg says.

Technicians brought the system down to install new memory. When that failed they reconfigured the system software to fix the memory problem, according to Osterberg. But then the machine started to process the backlogged mail and the system again became absurdly slow.

A day later, HASCS's technicians realized that part of the problem was being caused by the Eudora program. Although fewer students use Eudora, the way they used it was crippling the system.

Many Eudora users had the program set to automatically check their mail every minute or less, placing an overwhelming demand on the system.

"If you're a Eudora user and you're checking your mail every hour, we love you," Osterberg says. "If you're checking you're mail every minute, you're worse than a Pine user."

HASCS then revised the setup of the machines to only allow Eudora users to check their mail every 15 minutes.

As a result, some of the kinks have been ironed out.

"E-mail is, on the whole, stable," Osterberg said on Monday, although he added that there were still occasional problems with the load.

Obstacles

HASCS, which is charged with maintaining the computer network, has a difficult mission.

The organization has come a long way from three years ago, when then acting director Richard S. Steen (no relation to Frank) described HASCS in a confidential memo obtained then by The Crimson as "a ship taking on water faster than it can bail it out."

In the memo to the FAS IT Committee, then chaired by Martin and Lewis, Richard Steen cited three problems: budgetary constraints, staff and space shortages and poor working conditions.

The problems plaguing HASCS then have mostly been solved.

HASCS' budget has risen to $4.8 million for 1997, compared to $2.3 million in 1994, according to Martin.

And its staff has increased from 18 full time employees in 1994 to 32 today, according to Frank Steen.

"We have an adequate level of staffing and excellent people working for us," Steen says.

But HASCS now struggles with staff turnover--six people have left its UNIX group in the past two years, although one later returned. The UNIX group is responsible for maintaining and installing the servers that have been so problematic this year.

"Salary was not the main issue for any of the people who left," Steen says. "Most people left to pursue interesting jobs elsewhere."

However, jobs in HASCS pay significantly less than those in industry. And at least two of the six people to leave took jobs in industry.

The UNIX group now has six fulltime employees, who are on-call 24 hours a day. When the network crashed on Nov. 14, members of the UNIX staff worked most of the night to fix the problems, according to Osterberg.

The Crimson was unable to speak with any current UNIX staff members because Steen does not allow employees other than himself or Osterberg to speak with the press.

Solving the Problems

HASCS employees have been working feverishly to fix the problems with the system.

"Our staff has been working day and night to get the new equipment working," Steen says. "It took longer than we expected [but] the load was also greater than expected."

The IT committee also commended the job done by HASCS staff.

"The HASCS staff is a resilient group," reads the committee's September report. "They enjoy working hard and often work late at night and early in the morning when there would be little to see through their windows--if they had them."

Already this year, HASCS has made substantial improvements. It broke its own record for getting student accounts operating at the beginning of the year, by connecting 4,500 students in three weeks.

Computer support has also improved, according to Osterberg, who is responsible for coordinating the user assistants who help undergraduates and graduate students set up network connections and handle other e-mail and networking problems.

HASCS employs 52 students as UAs this year, compared to only 10 about five years ago, Osterberg says.

"Our residential support is on the cutting edge," Osterberg says. "Harvard's model is the second best in the nation only to Stanford which had a 10-year [lead]."

But HASCS will have to continue to work hard on the network because this fall's changes do not represent a long-term solution.

As a result of the increasing usage, Osterberg describes the newly, installed servers as "a medium term solution" which should last until the spring.

HASCS is considering major changes in the architecture of the e-mail system in order to achieve the expectation that "you should always be able to get your mail and always get it delivered immediately, you shouldn't have to wait to log-on and the system won't go down," Osterberg says. "That's the expectation we work to maintain."

The organization is considering instituting IMAP protocol which connects directly to the mail server but downloads only what a user needs, according to Osterberg.

IMAP, unlike Eudora, allows users to save their messages on a central server and retrieve them from any remote location.

Is There a Vision?

Despite the growth of its staff, HASCS's focus on simply keeping the system operating has prevented it from creating and following a larger vision for information technology in the College.

"Fixing the mail server has been very demanding work," Groppi says. "HASCS needs to be better supported by the University. There's a lot more stuff they could be doing if they had more resources."

The turnover in the UNIX group has left several projects undone and made the installation of the new FAS and mail servers particularly difficult.

Faculty and students have criticized HASCS for installing the new machines at the start of school, when the load increases the most. Installing the machines at the start of the summer would have been far more efficient, faculty and staff say.

But Steen says the machines were not installed earlier because the UNIX group had to stabilize before it could carry out the planning and installation.

The turnover has led to a loss of institutional memory and forced employees to acclimate themselves to Harvard's system.

"Turnover makes anything difficult," Osterberg says. "You lose your internal consistency when you bring new people in and they have to catch up and we have to get them used to our needs and our environment."

This turnover, and the unprecedented usage, has made coordinating ongoing projects and providing consistent service difficult.

"HASCS has to do a lot of firefighting," says Seltzer, a member of the IT committee.

For instance, HASCS has planned to work on authentication systems so that the College can provide grades and other sensitive material via the World Wide Web, but "we've had problems with UNIX staff turnover," Steen says.

HASCS has also worked on improving security on the FAS network but has thus far been unable to complete the job, according to Steen.

Presently when someone enters their password to log into their machine, it passes clearly over the network.

"With the right equipment, I could sit in the middle and watch your whole telnet connection with FAS, including what mail you send," Osterberg says.

HASCS's security efforts have emphasized to students the illegality of breaking into other people's accounts.

People who are caught are sent to the Administrative Board for disciplinary hearings, according to Steen. But he declined to say how many students are Ad Boarded each year because of computer violations.

The focus on keeping the system running also prevents HASCS from providing services including group accounts, mailing lists and Web pages to student groups. This service is instead handled by the Harvard Computer Society (HCS), which now serves 170 student organizations for free.

Groppi, who is HCS's director of account services, describes HASCS's refusal to help student groups as "unforgivable."

Groppi and HCS President Daniel A. Lopez '97 say HCS is happy to provide services to student groups and plans to continue its efforts.

But they say HASCS should be responsible for providing the services.

"We don't mind doing it, but Harvard should take a stand and make this an official service," Lopez says. "The need is big enough."Crimson File PhotoFRANKLIN M. STEEN, Director of HASCS